Llm decompilers sometimes hide errors despite passing tests

When LLM Decompilers Recompile More and Preserve Less

Cryptography and SecurityArtificial Intelligence

Summary

Decompilers try to turn machine code back into readable programs to help find bugs and understand software. The researchers found that AI-powered decompilers can produce code that looks perfect and even runs tests correctly, but still behaves differently in other situations, missing problems like security flaws. Traditional decompilers reveal unknown parts clearly, while AI versions replace them, hiding differences. To catch these hidden changes, the researchers created a new method that tests many inputs automatically to spot when decompiled code doesn't match the original behavior. Their method showed that even code passing all tests can differ in real use and sometimes hide serious errors.

decompilationmachine coderecompilationlarge language modelsvulnerability detectionmalware analysisfuzzingbehavioral testingsoftware securitycode equivalence

Authors

Chang Liu, Edward Raff, Kristopher Micinski

Abstract

Decompilation recovers high-level source from compiled machine code and serves as a foundation for security tasks such as vulnerability detection and malware analysis. Traditional decompilers like Ghidra and Hex-Rays expose whatever they cannot resolve as visible placeholders and often emit pseudocode that will not compile or execute; LLM-based decompilers produce clean, idiomatic C and are now judged almost entirely by recompilability and re-executability: whether the output builds and passes its shipped input/output tests. We show that these metrics can reward the wrong path: a function may recompile and pass every shipped test yet diverge on other legitimate inputs, and a disclosed vulnerability may disappear from the recompiled code with no visible trace of the crash. Neither failure is caught by existing suites. To address this gap, we propose Decompile-Diverge, a behavioral comparison oracle not relying on fixed or hand-crafted tests: for each function it synthesizes a driver, grows a fuzzing corpus from the reference, and reruns the decompiled code on the same inputs to detect changes in the function's behavior. Across eight systems in nine configurations on established LLM decompilation corpora, candidates that pass every shipped test still diverge from the original on our input corpus: 4.9% overall, and as many as 13% for a single system. On 300 real GitHub library functions and 287 CVE-grounded functions, recompilability and behavioral agreement can come apart: the strongest refinement LLM lifts Ghidra's build rate from 75% to 90%, while its Matched rate falls from 74% to 62%; on disclosed vulnerabilities, up to one tenth exhibit Crash Absence in its output. Source-level analysis traces this divergence to introduced fields, types, callees, and guards that replace the visible unknowns traditional tools leave behind.