Coding agents show limited success in closing chip timing issues
TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agents
Artificial Intelligence
Summary
Making computer chips work fast enough is a tough job that involves fixing timing problems in detailed electrical designs. The authors created TicTacBench, a test to see how well coding agents—software powered by advanced language models—can fix these timing problems in chip designs. They found that even the best agents could only fix a bit more than half of the timing issues, with some trade-offs in other design qualities. The authors also developed a method to guide agents step by step, improving their success rate but showing there is still lots of room for progress.
What this means in practice
- •For chip design engineers: Evaluate and improve automated tools for fixing timing problems in detailed chip designs with realistic constraints and reports.
- •For hardware verification teams: Use the benchmark to test coding agents’ ability to maintain functional correctness while optimizing timing in RTL designs after layout.
Authors
Bowei Wang, Zhigang Fang, Zhijie Yang, Renzhi Chen, Shanshan Li, Lei Wang
Abstract
Recent advances in large language models (LLMs) have led to the emergence of coding agents capable of performing complex engineering tasks, including register-transfer level (RTL) design and optimization. Existing RTL benchmarks mainly evaluate functional correctness and performance, power, and area (PPA) of the generated RTL designs, leaving agents' ability for \emph{timing closure} under-evaluated. We propose TicTacBench, a benchmark specifically designed to evaluate coding agents' capabilities for RTL-level timing closure under post-place-and-route (post-PnR) evaluation. TicTacBench contains 30 diverse tasks, each provided with a suboptimal RTL design, realistic timing constraints, functional equivalence verification, and timing reports. With over 300 runs of coding agents driven by 8 frontier LLMs, we find that even the best agent can only close 53.3\% of tasks with 7.18\% area-delay product (ADP) degradation and 8.83\% energy-delay-squared product (EDDP) improvement on average. We identify common failure categories that explain why agents fail to close timing. Then we propose TicTacSkill, a new method that guides agents to follow standard timing-closure procedures and improves the Timing Closure Rate by 9\%. These results suggest that while coding agents have made significant progress in RTL design, their timing-closure capability still has substantial room for improvement.