Large language models often suggest unneeded code changes with false speed claims

Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimization

Software EngineeringArtificial Intelligence

Summary

Large language models (LLMs) that help improve computer code sometimes make unnecessary changes to code that is already as fast as it can be, claiming these changes will make it faster even if they don't. The researchers call this problem Efficiency Hallucination, driven by a testing setup that pushes models to always edit code, even when it’s best to leave it alone. They created a new way to check when the model should not make changes, which helps the models learn when to stay quiet and only suggest improvements when needed. One of the newer models, GPT-5.4 Mini, almost perfectly knows when to avoid unnecessary edits, especially with simpler code. This approach can help make LLM tools safer and more reliable before they are used in real software development.

What this means in practice

  • For software development teams: Use the validation framework to reduce unnecessary code changes by AI helpers, improving trust and reliability in automated optimization tools.
  • For cloud service providers: Incorporate behavioral calibration methods to prevent AI-driven code updates that falsely claim performance gains, reducing deployment risks.

Authors

Sarah Wilson, Gail Kaiser, Patrick Musau

Abstract

The integration of Large Language Models (LLMs) into automated code optimization introduces a critical reliability risk we term the Efficiency Hallucination: an LLM's tendency to issue non-functional mutations with unsubstantiated performance claims on already-optimized code. This is driven by the Evaluation Trap, wherein binary benchmarks incentivize unnecessary modifications over safely abstaining. We present a validation framework using classification penalty methods, evaluated across 180 optimization runs on nine models (GPT, Claude, Gemini) using EffiBench. Under standard prompts, models exhibit a 100% over-edit rate on optimal code. Our guardrail raises correct abstention from 0% to to 44.4%, preserving a 100% edit rate on sub-optimal code with zero false abstentions. Calibration is uneven: GPT-5.4 Mini approaches near-perfect abstention, and simple code is recognized more reliably than complex code. Our framework offers a training-free mechanism to mitigate LLM overconfidence before deployment in production.
View PDFOpen arXiv