Code categories improve large language models differently by task
How code helps different tasks? A decompositional lens on LLM post-training
Artificial Intelligence
Summary
Choosing which kinds of code examples to use when improving large language models can change how well these models perform on different tasks like answering questions or writing code. The authors looked closely at different types of code solutions and how they affected models after training. They found that some code types help some tasks or models but can hurt others. Using small, carefully selected mixes of code types can even outperform training on all the code data at once, often with less data. This shows that more data is not always better—it depends on the mix and the task.
What this means in practice
- •For ai model developers: Improve model accuracy on tasks like math and question answering by fine-tuning with targeted code categories rather than full code datasets.
- •For software tool builders: Create more efficient AI training workflows by selecting compact code mixtures that boost performance using fewer data resources.
Authors
Zheng Yu, Yiwei Li, Yishen Chen, Xiang Li, Jiale Han, Benyou Wang, Jingbang Chen
Abstract
Evaluating code data as a single corpus can obscure which types of code data benefit which models and downstream tasks. Effective data selection requires understanding both the benefits of individual categories and whether these benefits persist when categories are combined. We introduce a decompositional lens for studying these effects in LLM post-training. We first decompose an execution-verified code corpus into interpretable categories based on the computational patterns of its solutions. Through controlled fine-tuning experiments, we compare individual categories with a balanced mixture across instruction-tuned models on question answering, mathematics, and code generation. The resulting response maps reveal recurring gains in average question-answering performance, while the same category can improve one model or task and degrade another. The best-performing category also varies with the starting model and target task. We then compose compact mixtures guided by these results and examine whether benefits observed in individual categories persist under joint training. On selected model--task pairs, mixtures whose constituents each improve the target task outperform both their best constituent and full-corpus training while using roughly 10--15\% of the full corpus. These exploratory findings illustrate a \emph{less is more} pattern and highlight how the value of code data in post training depends on which categories are combined for which model and task.