AI agents match humans in analyzing small software programs fast
Humans vs. Machines: A Comparison With AI Agents in Software Reverse Engineering
Software EngineeringArtificial Intelligence
Summary
Reversing software means understanding a program by looking at its code without having the original source. This study compares how well humans and AI models can analyze small software programs using a tool called IDA Pro. Some top AI agents perform just as well as skilled humans, but they use different strategies: humans carefully revisit code parts, while AI quickly grabs broad information once. While AI is fast and promising, handling larger software will need new ways to organize their analysis and remember past findings.
What this means in practice
- •For software security teams: Speed up identifying vulnerabilities in small programs by using AI agents that match human analysts in reverse engineering tasks.
- •For cybersecurity analysts: Improve analysis workflows by integrating AI agents that aggregate broad code information without repeatedly revisiting sections, enhancing efficiency on smaller binaries.
Authors
Samuele Doria, Stefano De Rosa, Ruoyu Wang, Zion Leonahenahe Basque, Eleonora Losiouk, Simone Aonzo
Abstract
Can an autonomous AI agent reverse engineer a binary as well as a skilled human analyst? Do they adopt the same strategy, and can agents scale to real-world programs? We compare human and LLM-based software reverse engineering (SRE): nine models tackle two compact static-analysis challenges previously solved by human analysts, scored with a common understanding rubric, and we record their complete interactions with IDA Pro. This lets us compare not only what humans and agents recover, but also how they investigate a binary. The best agents (GPT-5.5 Pro, Claude Opus 4.8, and Kimi K3) achieve statistically human-competitive understanding on both tasks, while weaker models remain far behind. Ablations show that the gap depends on code representation and workflow demands, not on model capability alone: open-weight models improve substantially with richer code representations, whereas frontier models show no significant change. Similar success rates hide different workflows. Humans repeatedly revisit functions and follow local program structure; strong agents jump across the binary, retrieve information broadly, and rarely return to inspected code, a behavior we call greedy fetching. This strategy is not fixed: given semantically richer code, agents fetch less without losing understanding. Because greedy fetching aggregates large amounts of code in the agent's context, we measure context-window saturation across 4,000 real-world binaries: large windows accommodate many programs, but larger applications and lower-level representations quickly exhaust them. Autonomous reverse engineering is becoming competitive, with a large speed advantage, but scaling it will require better navigation, selective retrieval, and persistent analysis state, not merely larger context windows.