Benchmark reveals large language models struggle with multi-turn code instructions
MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding
Computation and Language
Summary
When AI programs write code by following step-by-step instructions over multiple interactions, they often make mistakes or miss important details. The paper presents a new benchmark called MTAC-IFBench that tests how well these AI agents follow complex instructions when coding across many steps. This benchmark uses detailed checklists and automated checks to measure both if the code works and if it meets all the given requirements at each step. The authors found that current AI coding agents perform worse as the interaction continues, showing it is still hard for them to perfectly follow multi-turn instructions.
What this means in practice
- •For software development teams: Evaluate and improve AI coding assistants on following complex, multi-step programming instructions accurately across an entire development process.
- •For ai tool developers: Benchmark new AI models and agents for their ability to handle progressively detailed and constrained coding tasks in multi-turn interactions.
Authors
Bosi Wen, Cunxiang Wang, Jiayi Gui, Haoke Zhang, Yilin Niu, Pei Ke, Dayong Yang, Hongning Wang, Minlie Huang
Abstract
Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute, and utilize external tools iteratively to tackle complex tasks. Beyond achieving functional correctness, these agents must faithfully follow process instructions and constraints throughout the development lifecycle. However, existing benchmarks typically focus on final functional correctness or confine instruction-following evaluation to single-turn, general chat or simple code generation scenarios, leaving instruction-following in multi-turn agentic coding underexplored. To bridge this gap, we propose MTAC-IFBench, a comprehensive benchmark for this critical capability. It features multi-turn progressive software development instructions with diverse constraints spanning 6 primary and 18 secondary categories. With an average of 7.04 turns and 91.33 constraints per instance, it poses a rigorous challenge to current LLMs. To make the evaluation reliable, we construct a checklist for each constraint and functional requirement, and integrate verification scripts and judge agents to verify each checklist item. MTAC-IFBench identifies significant deficiencies in existing code agents in multi-turn instruction-following, with their performance degrading rapidly as the interaction session grows longer.