Runtime telemetry improves large language model web app generation success
TeleGen: Improving LLM-Based Web Application Generation via Runtime Telemetry
Software EngineeringArtificial Intelligence
Summary
Generating web applications using large language models can produce code that doesn't always work as expected when interacted with. The authors created TeleGen, a tool that adds monitoring inside these generated apps to record what happens during use, especially between user actions and final results. This detailed information helps pinpoint where problems occur and guides better fixes. Their tests show that using this runtime data notably improves how often generated web apps successfully complete their tasks.
What this means in practice
- •For web developers: Generate and iteratively improve web applications from natural-language specifications using telemetry-informed debugging.
- •For software testing teams: Diagnose subtle interaction failures in web applications by analyzing runtime telemetry between user actions and outcomes.
Authors
Yujia Luo, Haonan Zhang, Jiasi Shen, Zishuo Ding, Weiyi Shang
Abstract
Large language models can generate runnable web applications from natural-language requirements, but many generated applications still fail interactive tasks. Existing generate-execute-repair pipelines execute the generated application and use task outcomes or error messages to guide code revision. However, this feedback often misses the runtime behavior between a browser action and the final task outcome, making interaction-level failures difficult to diagnose. Therefore, we propose TeleGen, an observability-enhanced framework for LLM-based web application generation. TeleGen instruments generated applications, collects runtime telemetry during task execution, and compresses raw telemetry logs into concise briefs for repair. We evaluate TeleGen on WebGen-Bench and Web-Bench. On WebGen-Bench, TeleGen improves task success from 67.7% with repair without telemetry to 76.2%, an increase of 8.5 percentage points. On Web-Bench, it improves cumulative Pass@2 from 21.7% to 29.8%. Ablation results show that runtime telemetry provides a useful diagnostic signal, while telemetry briefs make this signal more effective and less costly to use. Further analysis shows that telemetry is especially helpful for failures involving hidden execution paths, such as navigation, form workflows, and frontend-backend coordination.