Fine tuning small ai models to predict low level code inputs
BUSTA: Teaching Small Language Models to Reason About Binary Execution
Cryptography and SecurityArtificial IntelligenceMachine LearningProgramming Languages
Summary
Reverse engineering is a process of understanding how a program works by analyzing its low-level instructions. The authors explored whether small language models can learn to reason about these instructions by predicting what inputs lead to desired outputs. They fine-tuned two smaller AI models to improve their accuracy significantly in this task. Although bigger models still perform best, the fine-tuned smaller models come close while requiring far less computing power. They also shared their fine-tuned models and training data for others to use.
What this means in practice
- •For software security engineers: Improve automated tools to analyze low-level code behavior in resource-limited environments by using smaller, fine-tuned language models.
- •For embedded systems developers: Use compact AI models to predict necessary inputs for assembly code segments when debugging or optimizing firmware.
Authors
Ritika Poddar, Aidan Fennelly, Deirdre Murphy, Eric Wolos, Corre L. Steele, Michael Doyle
Abstract
Large Language Models (LLM) are increasingly applied to Reverse Engineering (RE) workflows. RE, however, differs from common LLM coding tasks as it requires models to reason about low-level program behavior. Furthermore, the deployment of tooling in resource-constrained sandboxed RE environments limits the ability to use frontier-scale solutions. We examine whether a targeted LLM training approach can transfer distinct RE skills to open models via inverse basic-block execution: given an x86-64 basic block and a desired output register state, the model predicts the input register values that produce it. Fine-tuning \texttt{Qwen3-1.7B} and \texttt{Nemotron-3-Nano} with Group Relative Policy Optimization (GRPO) raises knowable-register prediction accuracy from 73.2\% to 85.3\% and from 63.3\% to 95.2\%, respectively. Frontier models still score highest (up to 99.2\%), but our fine-tuned \texttt{Nemotron-3-Nano} comes within four points at a fraction of the parameter count. We release both fine-tuned models and the training dataset.