A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions
2026-05-25 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors looked at many studies about using large language models (LLMs) to help generate computer code. They found that while these models do well on test problems, it’s less clear how well they work in real-world programming. They also noticed problems like how reliable and efficient these models are, and that issues like bias are not often studied. The authors suggest that future work should focus on making these models better suited for specific domains and developing more complete ways to evaluate them.
large language modelscode generationsoftware engineeringmodel evaluationbenchmarkingrobustnessefficiencybiassystematic reviewdomain adaptation
Authors
Muslim Chochlov, Michael English, Jim Buckley
Abstract
Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are promising, the broader effects of such application and their integration into real-world development remain insufficiently understood with existing tertiary studies provide little in this area. Objective. This tertiary study consolidates secondary evidence on LLM-based CGTs, synthesizing the publication landscape, effects, scenarios, integration challenges, and future research directions. Method. Following systematic review guidelines, we searched in related digital libraries, complemented by backward-and-forward snowballing and screening step. Study quality was assessed and extraction reliability was audited with inter-rater agreement statistics. Evidence was synthesized using SWEBOK knowledge areas and the HELM framework. Results. We identify 30 secondary studies published between 2017-2025, with rapid growth since 2023. Accuracy seems strong on benchmarks but weakly supported for real-world generalization; robustness is fragile across tasks and configurations; efficiency constraints are pervasive; toxicity and bias are under-reported. Dominant challenges concern economic feasibility, evaluation validity, and socio-technical integration. Future directions suggest domain-aware model improvement and the need for holistic, standardized evaluation. Conclusion. LLM-based CGTs represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domain-aware model improvements and holistic, standardized evaluation, addressing efficiency and associated costs.