CoSTAR enables smart COBOL code summaries using small AI models
CoSTAR: Data Synthesis-Driven Constraint-Aware COBOL Section Summarization for Legacy System Modernization
Software Engineering
Summary
COBOL is an old programming language still used by banks and governments, but it's hard to update because experts and documentation are scarce. The authors created CoSTAR, a system that makes helpful summaries of COBOL code sections by generating new example data and teaching smaller AI models to respect important rules. Their approach helps small AI models perform as well or better than much larger ones, which is useful for sensitive systems where privacy matters. This makes it easier to understand and modernize COBOL-based software.
What this means in practice
- •For enterprise software teams: Automatically generate accurate summaries of COBOL code sections to aid in understanding and modernizing legacy systems.
- •For privacy conscious system administrators: Use locally deployable smaller AI models for COBOL code summarization to ensure data privacy while maintaining high-quality results.
Authors
Hao Lin, He Jiang, Xiaochen Li, Weihong Sun, Yufu Wang, Zhilei Ren, Ang Jia
Abstract
COBOL remains critical to governments, financial institutions, and large enterprises; yet, aging technologies, shrinking expertise, and missing documentation make modernization of COBOL-based legacy systems increasingly urgent. Before migration, code summarization is a common practice to support legacy system understanding. However, COBOL code summarization, especially on section-level, faces two key challenges: data scarcity and migration constraint preservation. To address these challenges, we propose CoSTAR, an integrated framework that combines execution-validated data synthesis with constraint-aware model training. CoSTAR repurposes general-purpose programming tasks to synthesize execution-validated COBOL code-summary data through LLM-based generation to overcome data scarcity. Based on the synthesized data, CoSTAR augments target sections with relevant data declarations and natural-language explanations, and uses constraint-guided structured rationales to train smaller base LLMs. The trained LLMs preserve the migration constraints for COBOL section summarization. We evaluate CoSTAR on both public and confidential enterprise COBOL systems. CoSTAR effectively synthesizes 3,764 execution-validated training instances. Based on these instances, CoSTAR built on 7B/8B base LLMs can improve these LLMs with average relative gains of 25.38% on ROUGE-L, 53.84% on METEOR, and 37.22% on chrF. In real-world enterprise evaluation, CoSTAR built on only Qwen3-8B, outperforms the enterprise-deployed Qwen3-235B in accuracy, completeness, and conciseness. These results show that CoSTAR enables small, locally deployable LLMs to achieve performance competitive with substantially larger LLMs for privacy-sensitive COBOL legacy systems.