ProgramDistill evaluates coding agents by rebuilding web app features
ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
Software EngineeringArtificial Intelligence
Summary
It can be hard for coding agents to figure out what a web app is supposed to do when some parts are missing. This paper introduces ProgramDistill, a way to test these agents by having them rebuild parts of real web apps based on working examples. The authors created a method to break down apps into smaller parts with clear behaviors and challenges. They tested many coding agents and showed differences in how well they can handle tasks of varying difficulty. ProgramDistill helps measure and improve how coding agents learn to rebuild software correctly.
What this means in practice
- •For web developers: Evaluate coding assistants by testing their ability to restore missing parts of web apps based on existing working examples.
- •For software engineering teams: Diagnose strengths and weaknesses of AI coding tools in rebuilding software features to guide targeted improvements and training.
- •For automated code generation services: Develop more reliable AI programming assistants that incrementally reconstruct applications with verifiable behavior.$Commercial implications: Enables creation of AI coding products that improve software restoration accuracy for developers, making it marketable as enhanced programming support.
Authors
Jeonghye Kim, Minseon Kim, Young Jin Kim, Matheus Pereira, Marc-Alexandre Côté, Alessandro Sordoni, Xingdi Yuan, Zhengyan Shi
Abstract
Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications. We build ProgramDistill by factorizing applications into features of different granularities, each associated with replayable behaviors executable via its gold patch. Our pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention. Across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success falls from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8. ProgramDistill thus provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.