Generalist verifiers improve judgments on coding agent task success
SWE-Critix: Post-Training Generalist Verifiers for Assessing Coding-Agent Trajectories
Artificial Intelligence
Summary
Judging whether computer programs written by AI assistants succeed is hard because their work involves many steps and tools. The authors present SWE-Critix, a trained system that reads the entire work process and decides if the program successfully finished the task. It also explains its decision in plain language. SWE-Critix performs better than previous methods at correctly judging success, helping people trust and understand AI coding agents.
What this means in practice
- •For software development teams: Use SWE-Critix to automatically verify multi-step coding tasks performed by AI coding assistants, improving trust and reducing manual review effort.
- •For automated software testing teams: Integrate SWE-Critix to assess AI-driven code generation outcomes and receive explanations that highlight success or failure reasons during testing pipelines.
Authors
Jiajun Hu, Chun Yong Chong
Abstract
Coding agents are increasingly deployed for end-to-end software engineering tasks, where a readily available oracle evaluator that defines the success criteria is often absent. This motivates verifications for coding agents that can reliably judge and explain their behavior. However, verification is non-trivial as coding agents are increasingly capable of solving long-horizon software engineering tasks. Their execution trajectories can be extremely long, containing hundreds of iterative interactions with code repositories and external tools. Existing verifiers either rely on general-purpose LLMs without specialized training or train dedicated critics that predominantly focus on producing scalar rewards, highlighting the relatively limited exploration of training-based generalist verification. We introduce SWE-Critix, a post-training pipeline for training generalist verifiers of coding-agent trajectories. Given a task and its execution trajectory, the learned verifier predicts the completion label and generates a natural-language analysis that explains its judgment. Our pipeline combines supervised fine-tuning (SFT) with reinforcement learning (RL) and trains a series of 30B models on trajectories from CodeForge-Preview. On a held-out test set, SWE-Critix improves classification accuracy from 57% for the base Qwen3-30B-A3B reasoning model to 66% after SFT and 78% after RL. We release our models, datasets, and training code to facilitate further research.