Composing coding tasks boosts benchmarks and model training quality

CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

Machine LearningArtificial Intelligence

Summary

Collecting good coding problems is usually costly because it requires a lot of human work or complicated checking. The authors show that new, hard coding tasks can be created cheaply by combining simpler, verified problems in a careful way. They test this in different coding challenges and find that harder combined tasks give tougher tests for AI models and can help improve their training. This process doesn’t need expensive model-generated content or extra validation steps.

What this means in practice

  • For software testing teams: Create complex test cases automatically by combining simple verified coding problems to more thoroughly evaluate debugging and repair tools.
  • For ai model trainers: Generate challenging training datasets programmatically to improve the performance of coding AI models without relying on expensive human-curated examples.

Authors

Jiaxin Fang, Runyuan He, Arnav Gupta, Sahil Kumawat, Huanzhi Mao, Alvin Cheung

Abstract

High-quality coding problems are expensive to collect. Benchmarks such as SWE-bench rely on substantial human curation, while synthetic alternatives typically rely on costly model generation and validation. In this work, we show that new coding problems can be created at almost no cost by composing existing ones. We formulate problem generation as code composition: verified atomic problems are combined into level-$N$ tasks, and a task is kept only if every atomic problem remains unsolved and a joint reference solution passes the same check used at evaluation time. The entire task-construction process is programmatic and requires no language-model generation or validation. We instantiate this approach in four coding settings: multi-bug repair, proof completion, complexity analysis, and clone equivalence. The generated tasks are challenging for frontier models, which all degrade as compositional depth increases. The generated tasks are also useful as training data: rejection-sampling fine-tuning on them improves Qwen3.5-9B at every composition level.