Effect-aware evaluation reveals flaws in lLM multi-turn function calling
CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling
Artificial IntelligenceSoftware Engineering
Summary
Current evaluations of large language models (LLMs) that call external functions often miss when models perform unexpected or harmful actions during conversations. The authors improved an existing evaluation by tracking the effects of every function call, especially destructive ones, without replacing the original system. They found that many models perform destructive actions without asking for confirmation and that scoring based on these effects changes how model behavior is judged. This work shows the limits of current benchmarks in distinguishing careful clarification from proper permission-seeking during critical actions.
What this means in practice
- •For llm developers: Improve multi-turn function call evaluations by integrating effect tracking to reduce unsafe or unconfirmed actions in conversational agents.
- •For conversational ai designers: Design simulated users and scoring systems that better reflect the need for confirmation before potentially destructive commands.
Authors
Alexander Kovalenko, Artem Redchych
Abstract
The multi-turn checker of the Berkeley Function Calling Leaderboard (BFCL) evaluates final state correctness but ignores the effects of intermediate calls: extra calls are free, function documentation does not describe side effects, and the scripted user never responds to proposals. Consequently, models asking for confirmation before destructive actions are penalized; 260 of 734 reference turns in the base category perform destructive actions without confirmation. We make BFCL effect-aware without replacing it. We infer effect declarations for all 128 functions by executing them in BFCL simulators and comparing state changes, with destructive writes identified by three annotators. A judge-free replay scorer reproduces official results across all 800 reference trajectories while reporting unrequested destructive calls, duplicate writes, redundant reads, surplus steps, and their token costs. We also introduce a confirmation-aware protocol with a deterministic simulated user. Across 11 open models, effect-aware scoring preserves the official ranking but reveals substantial differences in execution quality: the leading model loses 14.0 points due to unrequested destructive calls, while surplus steps consume up to 28% of input tokens. Adding effect hints to function documentation has no impact on eight evaluated models. Allowing the simulated user to respond recovers up to four points for models that seek confirmation. However, the best model completes only six of 576 conversations requiring destructive actions with explicit confirmation. These results expose a key limitation of current function-calling benchmarks: they cannot distinguish between asking for clarification and obtaining permission before potentially destructive actions.