Syntax-aware tuning improves code-focused language models performance
SynMOE: A Syntax-Aware MOE-Guided Approach for PEFT-Based Code Intelligence Learning
Software Engineering
Summary
Fine-tuning language models to understand computer code often requires adjusting many internal settings, which can be slow and costly. The authors found that current methods for more efficient tuning don't perform as well on code tasks because they don't consider the structure of the code. They created SynMOE, a way to include code syntax information when fine-tuning, which helps the models work better with less computing effort. Tests show SynMOE improves results consistently and even matches or beats full fine-tuning in some cases.
What this means in practice
- •For software engineering teams: Enhance code analysis and completion tools by fine-tuning language models with syntax-aware adaptations to boost accuracy without full retraining.
- •For cloud platform providers: Offer more efficient code intelligence services by deploying syntax-aware fine-tuned models that require less computation while maintaining high performance.
Authors
Shuo Liu, Hao Ning, Jacky Keung, Zhen Yang, Xiao Yu, Xing Hu, Yicheng Sun
Abstract
Parameter-Efficient Fine-Tuning (PEFT) has emerged as an important approach for adapting Language Models (LMs) in downstream tasks. However, in the domain of code intelligence, the performance of most PEFT methods relatively drops compared to Full fine-Tuning (FT), hindering their broader applicability. One of the main reasons is that current PEFT methods are generally based on assumptions about the low-rank nature or sparsity of parameter updates, lacking explicit modeling of code-related information. To tackle this limitation, we propose SynMOE, a syntax-aware MOE-guided approach for PEFT on code intelligence learning. Specifically, SynMOE contains two stages: syntactic information extraction and syntax-aware MOE-guided adaptation. It first obtains the syntactic labels of source code from Abstract Syntax Trees (ASTs), and then aligns the labels with the input tokens of code LMs. In the second stage, syntactic labels are integrated into the MOE module, participating in the routing decision of MOE for a more powerful syntactic-aware token representation. We evaluate the effectiveness of the proposed approach by applying it to two typical PEFT methods, LoRA and $(IA)^3$, across three diverse code intelligence tasks. Experimental results show that SynMOE achieves consistent and significant improvements. Even compared to FT, SynMOE achieves comparable or superior performance while being more computationally efficient. In addition, we also conduct ablation studies, hyperparameter analysis, and evaluation in the low-resource scenario to provide a comprehensive investigation of the proposed approach.