Language agents tested on selective action suspension and resumption
NAQD Env: A benchmark for selective withdrawal in language agents
Artificial Intelligence
Summary
When language-based AI agents are interrupted or told to stop, they need to decide which tasks to pause, which to continue, and when to resume. The authors created a simulated environment called NAQD-Env to test how well these agents follow such instructions. They found that current AI models rarely pause or resume tasks correctly as expected and often fail to match ideal decision-making. Their experiments show the need to treat selective stopping as a key part of making reliable AI agents.
What this means in practice
- •For language model developers: Evaluate and improve how language agents handle interruptions and resume tasks using NAQD-Env benchmark scenarios.
- •For ai system testers: Use the environment to detect unsafe or incorrect withdrawal behaviors in instruction-tuned models before deployment.
Authors
Mohamed Abouzahra
Abstract
Language agents must revise planned actions when evidence changes, permission is revoked, or a stop instruction arrives. A useful response is selective: suspend affected actions, preserve unaffected work, and resume only after sufficient repair. We introduce NAQD-Env, a synthetic environment that evaluates these decisions against a deterministic reference policy over explicit evidence, authorization, and constraint dependencies. Eleven dependency families support evaluation on development structures, held-out families, and held-out combinations of structures. Metrics distinguish attempted violations from violations permitted by a simulated execution gate and jointly report policy agreement, task value, withdrawal, resumption, and event reporting. We evaluate three open-weight instruction-tuned models from two families under three prompt conditions on 350 frozen scenarios, yielding 3,150 model-prompt episodes before gate replay. Across the reported conditions, withdrawal recall is at most 0.06, no valid resumption is observed at eligible opportunities, and only one episode matches the complete reference policy. Under the NAQD prompt, Qwen2.5-7B has fewer unsafe-attempt episodes than Qwen2.5-3B and Llama-3.1-8B, but also completes less useful work and preserves unaffected actions less accurately. Exploratory supervised fine-tuning probes increase Qwen2.5-3B decision accuracy from 0.45-0.54 to 0.83-0.92; separate diagnostics reveal inappropriate withdrawal after curriculum omissions and a loss of event reporting. These results motivate evaluating selective withdrawal as a distinct component of agent reliability. The setting measures policy application with trusted structured inputs and does not establish real-world containment or source-verification ability.