Human feedback reveals specific internal errors in llm agent decisions
Human Interventions as Labels: Localizing, Steering, and (Sometimes) Editing the Computations Behind LLM Agent Failures
Machine LearningArtificial IntelligenceComputation and Language
Summary
When AI language agents make mistakes, humans usually correct them, but these corrections are often ignored beyond training. The authors look at these human fixes as clues to where exactly inside the AI the problem happened. They find that for some AI models, they can pinpoint and even change the exact step that causes the AI to ask unnecessary questions instead of acting. This approach helps reduce extra human checking without losing important questions, and refining the AI can be done by carefully adjusting internal parts rather than just copying behavior.
What this means in practice
- •For llm developers: Pinpoint and correct internal computation steps causing unnecessary questions in language model agents to improve efficiency.
- •For ai system operators: Use human feedback as direct signals to steer LLM agent behavior without increasing oversight workload.
Authors
Cesare Zavattari, Alessandro Tommasi, Giuseppe Prencipe
Abstract
When an LLM agent misbehaves, a human steps in, and that intervention is usually discarded or replayed as training data. We treat it instead as a measurement: a label pointing at the internal computation that failed. We study it in a controlled agent environment, on one failure (asking a clarifying question when the request was already unambiguous), with labels from a rule-based overseer validated against blind human judgment. Claims cover eligible models, which both over-ask and sometimes act on matched unambiguous requests: five of thirteen qualified (Qwen2.5-3B, gemma-2-2b, OLMo-3-7B, Mistral-Nemo-12B, Phi-4). Four findings. First, the ask-vs-act decision is causally localizable in every eligible family: a direction learned from intervention labels flips 72-100% of held-out over-asks at a definite residual-stream depth, with site and dose chosen on training data, and random directions under the same procedure flip at most one episode. Second, over-asking has two mechanistic signatures: in four families the model demonstrably knows the request is unambiguous (the belief is decodable, causally inert and orthogonal to the decision) and asks anyway; in the fifth, the rare over-asks track different world-representations. Third, steering gated on a probe readout removes most of the simulated oversight burden while legitimate questions survive (bootstrap 95% intervals exclude zero); a prompt instruction makes it worse. Fourth, consolidating the fix into the weights yields a map, not a rule: on one model a LoRA confined to the causal band passes every criterion fixed in advance while behavior cloning on identical data breaks retention; on three families the reverse holds; on the fifth the outcome crosses the boundary across seeds. A geometric rule written down in advance to predict the regime failed on two out-of-family tests; we report the failure and stop rather than fit a replacement.