Decompiler tool recovers clear code from virtual machine obfuscation
VMSpec: A Decompiler-based Approach against Virtual Machine Obfuscation
Cryptography and Security
Summary
Some software hides its instructions by running on a made-up mini computer inside the real one, making it hard to understand. The authors present VMSpec, a tool that can automatically figure out how this mini computer works and turn the hidden instructions back into clear code. This helps security experts see what the hidden programs really do, even if they use complex tricks to confuse analysis. VMSpec works well on many test programs and real-world malware samples, matching the clarity of normal decompilers.
What this means in practice
- •For malware analysts: Automatically convert VM-obfuscated malware into readable code for faster behavior understanding and threat assessment.
- •For software security teams: Analyze obfuscated internal code by recovering high-level program logic for vulnerability detection and auditing.
Authors
Fabian Kilger, Leonard Auer, Manuel Andreas
Abstract
Malicious actors frequently obfuscate their code to hinder investigative reverse engineering efforts. Among the most formidable techniques is virtual machine (VM) obfuscation: programs are translated into bytecode for a custom VM, often with a randomized virtual instruction set and distributed alongside a corresponding interpreter. Manual analysis and deobfuscation scale poorly due to the complexity of VM obfuscation and the diversification techniques that render each VM and interpreter unique. Existing methods primarily focus on simplifying execution traces and identifying virtual execution boundaries. However, comparatively little attention has been given to recovering higher-level program semantics from VM-obfuscated code. To address this gap, we propose VMSpec, a novel decompiler-based approach that automates the construction of VM processor specifications, enabling existing, well-known decompilers to analyze the VM-obfuscated code. We evaluate our prototype of VMSpec on 147 programs from an established obfuscation benchmark, comprising hash functions, standard algorithms, and randomized functions, obfuscated with different configurations of the Tigress and Code Virtualizer obfuscators. To demonstrate real-world applicability, we additionally include six domain generation algorithms stemming from real-world malware. VMSpec completely strips VM-obfuscation from nearly all obfuscated samples, producing output comparable to Ghidra's decompilation of the original code. Furthermore, it is the first method capable of reconstructing the complete program behavior of VM-obfuscated programs with infinite, input-dependent paths and recursive calls.