SEMPRE improves understanding of complex communication protocols automatically
SEMPRE: Semantic-Aware Protocol Reverse Engineering via Consistency-Driven Joint Optimization
Cryptography and Security
Summary
Understanding unknown communication formats is important for security but hard because current methods rely on fragile rules and struggle with complex nested messages. The authors propose SEMPRE, a new tool that uses AI language models combined with dynamic code analysis to automatically break down and interpret these messages. SEMPRE identifies the different parts of a communication, including nested sections, and improves its guesses step-by-step until it gets accurate results. Tests show SEMPRE performs much better than existing methods on common protocols and real-world malware traffic.
What this means in practice
- •For network security teams: Automatically identify unknown protocol messages to improve threat detection and forensic analysis.
- •For malware analysts: Decode complex command-and-control communications from malware without manual protocol knowledge.
Authors
Yihua Xu, Weijie Huang, Xiaofeng Wang, Hang Zhang, Binghui Wang, Yue Duan
Abstract
Protocol reverse engineering (PRE) is essential for understanding undocumented or proprietary communication formats, enabling security-critical tasks such as vulnerability discovery, malware C2 analysis, and IDS signature generation. However, existing PRE techniques rely on brittle hand-written rules, under-model contextual dependencies among neighboring fields, and often fail on messages with nested sub-protocols. We present SEMPRE, a fully automated PRE framework that replaces ad-hoc heuristics with recursive LLM reasoning grounded in fine-grained dynamic program analysis. SEMPRE uses taint-flow and loop traces to guide an LLM-driven tree-of-thought search that slices messages into fields while detecting nested sub-payloads, producing a hierarchical layout. It then infers each field's type and role and iteratively refines boundaries by freezing high-confidence segments and re-running detection under semantic constraints, yielding an EM-style procedure that monotonically improves structure and semantics until convergence. Evaluated on a broad corpus of mainstream protocols against six state-of-the-art (vs. <0.4 for all baselines) and 0.72 for semantic inference (vs. 0.13 for BinPRE). Ablations confirm the contribution of each component, and a case study on a prolific real-world botnet demonstrates practical utility.