HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?
2026-04-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors explain that advanced coding agents struggle not because they lack skill, but because they can't always judge when to ask for help versus when to act on their own. Current tests don’t catch this problem because they give clear instructions and only check if the final answer is right. To fix this, the authors created HiL-Bench, a test that includes tricky parts needing clarification, and a score (Ask-F1) that measures how well an agent knows when to ask questions. They found most models do poorly at this help-seeking skill but showed it can improve when trained to recognize uncertainty. This improvement also works across different task types.
coding agentshelp-seekingambiguitybenchmarkAsk-F1uncertainty detectionreinforcement learningtask performancehuman-in-the-loopjudgment
Authors
Mohamed Elfeki, Tu Trinh, Kelvin Luu, Guangze Luo, Nathan Hunt, Ernesto Montoya, Nandan Marwaha, Yannis He, Charles Wang, Fernando Crabedo, Alessa Castilo, Bing Liu
Abstract
Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but judgment: knowing when to act autonomously and when to ask for help. Current benchmarks are blind to this failure mode. They supply unambiguous detailed instructions and solely reward execution correctness, so an agent that makes a lucky guess for a missing requirement will score identically to one that would have asked to be certain. We present HiL-Bench (Human-in-the-Loop Benchmark) to measure this selective escalation skill. Each task contains human-validated blockers (missing information, ambiguous requests, contradictory information) that surface only through progressive exploration, not upfront inspection. Our core metric, Ask-F1, the harmonic mean of question precision and blocker recall, captures the tension between over-asking and silent guessing; its structure architecturally prevents gaming through question spam. Evaluation across SWE and text-to-SQL domains reveals a large universal judgment gap: no frontier model recovers more than a fraction of its full-information performance when deciding whether to ask. Failure analysis identifies three key help-seeking patterns: overconfident wrong beliefs with no gap detection; high uncertainty detection yet persistent errors; broad, imprecise escalation without self-correction. These consistent patterns confirm poor help-seeking is a model-level flaw, not task-specific. RL training on shaped Ask-F1 reward shows judgment is trainable: a 32B model improves both help-seeking quality and task pass rate, with gains that transfer across domains. The model does not learn domain-specific heuristics for when to ask; it learns to detect unresolvable uncertainty and act on it.