LLM models agree to score COBOL code summaries without references
Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization
Artificial IntelligenceProgramming Languages
Summary
When computers summarize code, it’s hard to check if the summary is correct without someone’s help. The authors found that if different large language models (LLMs) agree on what a summary means, it’s probably true. They use one LLM to act like a judge, comparing summaries two at a time to decide which is better. Then they combine these judgments carefully to decide which summaries are likely right, even without any human-written example to compare against.
What this means in practice
- •For software maintainers: Assess the reliability of automated summaries for legacy COBOL code without needing expert reviews for each case.
- •For quality assurance teams: Automatically gauge the correctness of code documentations generated by different AI tools to focus manual checking efforts.
Authors
Marco Colombo, Balaji A Jayaram, Sowmyakumari Selvaraj, Sumit Tokkar, Animesh Kumar, Aravinda Prasad Sistla, Ravishankar Shetty, Rigel Gjomemo
Abstract
This paper describes an approach and use case to estimate the correctness of LLM-generated COBOL summaries without access to human-authored references. The core intuition is that inter-model agreement on semantics is informative about truth, and that an LLM-as-judge can provide higher-fidelity pairwise assessments than traditional string- or embedding-based metrics. We operationalise this intuition with (i) controlled summary generation from diverse models; (ii) a LLM evaluator that compares pairs of summaries and emits structured judgments; and (iii) a calibrated aggregation rule that combines supportive evidence while penalising only corroborated contradictions.