IBBench-Light tests model responses to matched task instructions
IBBench-Light: A Paired Evaluation of Task-Conditioned Responses to External Directives
Artificial IntelligenceLogic in Computer Science
Summary
Understanding how AI models handle different instructions for the same task is tricky because average scores can hide details. The authors created IBBench-Light, a benchmark that pairs two related instructions using the same input to see if models meet both output requirements exactly. They tested several models and found that success rates drop when evaluated on paired tasks compared to individual ones. They also explored how changes in stopping criteria affect performance, showing that evaluation details matter when assessing AI instruction-following capabilities.
What this means in practice
- •For ai developers: Improve evaluation of language models by testing paired task instructions to better understand output reliability across related prompts.
- •For quality assurance teams: Design testing protocols that measure consistent model performance on closely related task instructions using exact output contracts.
Authors
Kainan Zhou, Gangzhen Qian, Zhaoyi Li, Hang Xiao
Abstract
An external record may contain a procedure to apply or text to read, depending on the user's request. IBBench-Light tests both uses against the same record. Twelve semantic bases yield 144 matched pairs per model; four quantized instruction models produced 1,152 archived greedy responses. Paired exact-contract accuracy (PECA) requires both members to satisfy their output contracts. Qwen succeeds on 132 execute and 109 process prompts, but only 97 complete pairs, showing what marginal averages omit. We audit literal-target exposure and case normalization, then add 1,722 logged CPU generations to test directive-absent controls, twelve additional semantic bases, within-base wording changes, and generation stopping. In the pinned Phi rerun, changing the end-of-sequence (EOS) set changes exact paired success from 0/144 to 62/144. A bounded IHEval comparison uses the same SmolLM2 checkpoint and output budget while preserving its published instruction roles and scorer. The benchmark measures conditional task and output-contract success. Its task margins and paired count need to be read together with the stopping policy.