Confidence-guided method improves testing for AI code ranking
Confidence-Gated Transductive Test Generation for Code Reranking
Artificial IntelligenceComputation and LanguageSoftware Engineering
Summary
It is hard to create good test examples to check if computer programs generated by AI are correct, especially when expected answers are unknown. The authors developed a new method called CoTT that first tries a quick approach and only does extra checking when it isn't confident enough. This saves time while still making sure the test results are trustworthy. Their tests show CoTT works better and faster than older methods at ranking AI-generated programs.
What this means in practice
- •For software development teams: Improve automated evaluation and ranking of AI-generated code efficiently in development workflows.
- •For quality assurance engineers: Reduce costs of generating and validating test cases for code produced by large language models.
Authors
Sungjae Lee, Youngsik Yoon, Seockbean Song, Siwei Wang, Wei Chen, Jungseul Ok
Abstract
Test case synthesis is crucial for evaluating and ranking programs generated by large language models (LLMs). However, constructing high-quality test cases remains challenging because reliable expected outputs are often difficult to obtain. We propose Confidence-Gated Transductive Test Generation (CoTT), which first uses an efficient inductive procedure and invokes transductive generation only when inductive confidence is low. This adaptive design improves output reliability while allocating extra computation only when needed. On code reranking benchmarks, CoTT outperforms prior baselines across the reported metrics while reducing cost relative to applying transductive generation to every input. These results show that confidence-based allocation of test-time computation provides a favorable efficiency-effectiveness trade-off with a single efficient LLM.