Code models compared fairly to find best training methods
Comparing the Pre-trained Models of Source Code by Re-pre-training under a Unified Setup
Software Engineering
Summary
It can be hard to tell which computer models that understand programming code work best because they are tested in different ways. The authors looked at many existing code models and tested them again in the exact same way. This helped show how different training methods affect how well the models do on software engineering tasks. Their work helps others know which approaches work better and how to improve code models in the future.
What this means in practice
- •For software engineering teams: Choose the best pre-training methods to improve code analysis and understanding tools used in development workflows.
- •For programming tool developers: Develop code completion and bug detection features with models selected for their strengths on relevant tasks.
Authors
Changan Niu, Chuanyi Li, Vincent Ng, Bin Luo
Abstract
Recent years have seen the successful application of large pre-trained models of source code (CodePTMs) to code representation learning, which have taken the field of software engineering (SE) from task-specific solutions to task-agnostic generic models. By the remarkable results, CodePTMs are seen as a promising direction in both academia and industry. While a number of CodePTMs have been proposed, they are often not directly comparable because they differ in experimental setups such as pre-training dataset, model size, evaluation tasks and datasets, etc. In this paper, we first review the experimental setup used in previous work and propose a standardized setup to facilitate fair comparisons among CodePTMs to explore the impacts of their pre-training tasks. Then, under the standardized setup, we re-pre-train CodePTMs using the same model architecture, input modalities, and pretraining tasks, etc., as they declared and fine-tune each model on each evaluation SE task for evaluating. Finally, we present the experimental results and make a comprehensive discussion on the relative strength and weakness of different pre-training tasks w.r.t. each SE task. We hope our view can inspire and advance the future study of more powerful CodePTMs.