Quantization changes how coding AI solves problems without losing accuracy
Beyond Accuracy: How Quantization Affects Coding Agents
Computation and LanguageArtificial Intelligence
Summary
Using smaller, more efficient versions of big coding AI models does not lower their overall accuracy, according to the authors. However, these smaller models tend to solve problems in very different ways, such as using loops instead of one-shot solutions or relying more on external tools. The authors also found that types of errors made by the models changed depending on size and quantization method. This suggests evaluating these AI coders needs more than just checking if answers are right or wrong.
What this means in practice
- •For software engineering teams: Optimize deployment of coding AI by understanding how quantization affects their problem-solving behaviors and error patterns beyond accuracy.
- •For cloud service operators: Manage resource allocation for coding AI services by assessing tool usage and failure modes under quantization to improve reliability and cost efficiency.
Authors
Areej Mehboob, Muhammad Ali Shafique, Muhammad Usman Qadeer, Hamza Farooq
Abstract
Agents are increasingly used for coding tasks, but the inference cost of large language models (LLMs) limits where they can run. Quantization reduces model size, but its effects on coding agents that execute and revise programs remain underexplored. We compare BF16 with 4-bit AWQ and GPTQ for Qwen2.5-Coder-Instruct (7B, 14B, and 32B) and Devstral-Small-2507 (24B) on OJBench coding problems across three seeds. We detect no statistically significant accuracy loss for any of the coding agents. However, final accuracy can conceal substantial behavioral changes: AWQ reduces Qwen-7B one-shot solves from 17 to 0 while increasing loop-based solves from 10 to 25. All Qwen-32B successes involve tool use, whereas Devstral rarely makes a valid executor call. Failure profiles also differ: Qwen-32B produces far more compilation errors than the smaller Qwen models, rising slightly under quantization, while Devstral's failures shift significantly from runtime errors toward wrong answers and timeouts. These findings indicate that quantization robustness in coding agents should be assessed through solution trajectories, tool use, and failure modes, in addition to final accuracy.