A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents
Artificial IntelligenceSoftware Engineering
Summary
The authors studied how the quality and amount of training examples (trajectories) affect the fine-tuning of a code-writing AI model called Qwen2.5-Coder-7B-Instruct. They created a way to score trajectory quality based on Efficiency and Style, then tested different data selections. They found that when using small amounts of data, simply increasing the quantity helps more than choosing only the highest-quality examples, but with more data, quality becomes more important. Their work suggests it’s useful to score and filter training data for improving these code agents, especially as dataset size grows. They also proposed a practical evaluation method when direct task success measurement is too hard.
Authors
Yunze Han
Abstract
Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models. A central yet underexplored question is how trajectory quality and quantity jointly shape model performance. We present a systematic empirical study of trajectory data filtering for LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct on the SWE-trajectory dataset (67,074 trajectories, of which 32,161 are resolved). We propose a two-axis quality scoring framework -- Efficiency and Style -- and evaluate it through 16 controlled experiments spanning strategy, scale, and ablation analyses. Since 7B-scale models attain near-zero SWE-bench resolve rates, we adopt cross-entropy (CE) loss on held-out trajectories as the primary metric, validated via first-action generation: CE loss and ROUGE-L are perfectly rank-correlated (Spearman $ρ$ = -1.00), with limited-sample evidence supporting but not conclusively establishing this proxy. Our results reveal a scale-dependent quality-quantity trade-off: at small scales, doubling the dataset (500 to 1,000) yields ~12.7% CE-loss reduction whereas the TopQ-Random gap stays <1% (Mann-Whitney p > 0.10); at 2,000 trajectories this same gap widens to 3.6% (p = 0.016). Ablation further identifies error-retry rate as the dominant sub-dimension, performing comparably to the full composite ($Δ$ < 0.2%). Together, these findings establish trajectory-level quality scoring as a viable but scale-sensitive lever for code-agent SFT and offer a proxy-validated evaluation protocol for the regime where end-to-end resolve rate is statistically infeasible.