Cost-efficient large language models struggle to produce correct enterprise code
An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks
Software EngineeringArtificial Intelligence
Summary
This paper tests three affordable large language models to see if they can write accurate software code from instructions. The authors gave the models almost 1,000 programming tasks in Java and checked the results carefully. While most code looked like it fit the rules, many did not actually work properly or gave wrong answers. The study shows these models often guess answers rather than truly compute them, and correct solutions were quite rare. The authors also note some limits to their testing approach.
What this means in practice
- •For software engineering teams: Assess cost-efficient language models before using them to generate enterprise code, to avoid relying on incorrect or unreliable outputs.
- •For software testing teams: Improve testing strategies for code produced by language models by understanding frequent failure modes and response patterns revealed in this study.
Authors
Chandimal Adikari, Nandika Herath
Abstract
This study empirically evaluates whether cost-efficient Large Language Models (LLMs) can be trusted to generate enterprise code to a written specification. Three models (Gemini Flash 3, GPT-5.4 mini and Claude Haiku 4.5) were asked to solve 992 algorithmic problems as Java Spring Boot service methods conforming to a mandated signature and data-transfer-object specification, crossing four model and agentic coding tool combinations with two prompt variants to yield eight configurations, with iteration forbidden and hardcoded answers explicitly prohibited. Eight problem statements were withheld to probe how models respond to missing input. The 7,593 resulting methods were classified by an eight-class outcome taxonomy describing what each does about producing an answer, then deployed and executed, giving 7,936 measured requests joined to that classification. Structural conformance approached ceiling, yet 38.4% of methods do not compute the value they returned and only 12.9% of returned answers were correct. Conditioning on outcome class shows that response reliability and correctness are inversely related, whereas genuinely computing methods answered least often and were correct 19.3%. Limitations include single generation runs per configuration, partial harness coverage, single-pass timing, syntactic classification, and probable corpus contamination.