Memorization floor reveals limits of ai code refinement gains
A Memorization Floor for LLM Refinement of Decompiled Code
Software Engineering
Summary
Improving decompiled computer code using large language models (LLMs) may seem promising, but the authors found that much of what these models add comes from memorizing patterns rather than understanding the input code. By destroying the input's identifying features and testing the model’s ability to recover names, they showed that the improvements don't rely on the input itself. This suggests that current LLM refinements of decompiled code might not truly understand or improve the code as much as hoped. The study carefully tested multiple models and conditions, but no strong benefits based on input code were confirmed.
What this means in practice
- •For software security teams: Evaluate if AI-assisted code analysis tools actually improve understanding of decompiled code or just replicate memorized patterns.
- •For software reverse engineers: Avoid relying solely on LLM-based refiners for accurate identifier recovery in decompiled functions, as improvements may not reflect genuine input understanding.
Authors
Muhammad Asjad
Abstract
We introduce a memorization floor: a within-item control separating what LLM refinement of decompiler output recovers from its input from what it recovers from its prior. Refine a function, then refine it again from an input whose identifiers have been destroyed, and measure what survives. Because the comparison is within-item, corpus difficulty cannot contribute; it costs twenty API calls. Applied to functions written after our analysis plan was committed, so no released model could have memorized them, it reports two things. Recovery is real: refined output sits +0.072 to +0.137 above an arm-matched permutation null built from its own output vocabulary. But it does not depend on the input we ablate: destroying the input's dataflow changes the naming gain by +0.001 (95% CI [-0.026, +0.026]), and removing type prefixes or permuting names changes it by no more. A second refiner from another vendor, registered in advance and given byte-identical inputs, reproduces this -- twelve contrasts, two models, twelve nulls. Readability stays at ceiling throughout, so a reader is given no signal. The null is bounded, not absolute: contributions under 0.056 are invisible, and the ablation leaves operations intact, so naming from those alone remains a competing reading. No registered hypothesis was confirmed, and we report the five instrument failures behind that in full, including a reassembly harness biased against the treated arm and an equivalence checker we registered without checking it worked on our inputs.