LLMs generate code confidently even when execution fails often

Characterizing Overconfident Failure in LLM-Based Code Generation

Software EngineeringArtificial Intelligence

Summary

Large language models used to write computer code sometimes produce programs that look right but don't actually work. The authors found that these models often give high confidence to wrong codes, making it hard to tell correct from incorrect just by looking at their confidence scores. They tested different methods to spot or reduce these overconfident errors but found these approaches only partially help. Interestingly, the authors suggest that hidden information inside the model might hold clues about correctness that current confidence measures miss.

What this means in practice

  • For software developers: Use insights on overconfidence to better evaluate automated code outputs before execution tests.
  • For ai tool builders: Improve code generation tools by exploring latent model signals to detect failures beyond confidence scores.

Authors

Ravishka Rathnasuriya, Wei Yang

Abstract

Large language models (LLMs) are increasingly used for automated code generation, but generated programs can appear syntactically plausible while still failing execution-based correctness checks. Existing validation methods, such as testing and program analysis, remain essential but are often incomplete, costly, or applied only after generation. Model-derived uncertainty is therefore a natural early reliability signal. This paper studies the dilemma of overconfidence in code LLMs where incorrect programs are often generated with token-level confidence comparable to correct programs. We study this dilemma across four open-source code models and three execution-based benchmarks. Our analysis begins by investigating whether existing uncertainty metrics provide reliable proxies for execution correctness in code generation. We then characterize overconfidence at both global and local token levels, asking whether incorrect programs remain indistinguishable from correct ones under confidence and entropy summaries, including selective generation and the limits of instruction tuning. Finally, we evaluate whether common mitigation strategies reduce this failure mode. Our study yields four findings. First, existing uncertainty signals provide only partial and model-dependent evidence of execution failure. Second, overconfidence persists at both program and token levels, and uncertainty-based selection does not consistently improve accepted-set accuracy. Third, instruction tuning can increase certainty on failing generations without consistently improving correctness discrimination. Fourth, common mitigation techniques improve specific aspects of reliability but do not reliably resolve overconfident failure. Our exploratory latent analysis suggests that hidden representations may encode correctness-related signals that output confidence does not expose.