PresenceBench measures AI response to users’ mental states during tasks
PresenceBench: An Exploratory Evaluation of Large Language Model Adaptation to User State
Human-Computer InteractionArtificial IntelligenceComputation and Language
Summary
AI systems sometimes try to adjust how they respond based on how ready or focused a person is. This paper introduces PresenceBench, a way to measure if AI models really improve their replies when given information about a person's mental state. The authors tested four AI models on different tasks to see if they respond better with user state info versus without. They found some signs of improvement, but it wasn’t always clear if the AI was truly responding to the right mental state or just changing because it got extra instructions. The tool and their initial results provide a starting point for future tests.
What this means in practice
- •For chatbot developers: Evaluate if chatbots can adjust their replies based on users’ mental states to improve interaction quality without reducing safety.
- •For customer support platform teams: Use PresenceBench to test if AI helpers in support centers respond better when given signals about user readiness or frustration.
Authors
Peña Arroyo Gallego, Mauro Ruiz Moreno, Alejandro Acien, Sergio Romero-Tapiador, Teresa Arroyo-Gallego
Abstract
Presence-aware intelligence is the capacity of an AI system's intelligence layer to adapt appropriately to the functional state of the person it serves (here readiness, flow, and stability) while preserving task quality and safety. We introduce PresenceBench, an instrument for measuring it. PresenceBench evaluates how the intelligence layer uses a supplied state, not how that state is sensed, and breaks the capability into measurable properties. Two are tested here. Receptivity: does the model take up the state and give a better response than with no signal? Specificity: does any improvement depend on the correct state, rather than on extra structured context? A third, adaptation across states, is a required endpoint not measured in this release. A five-arm causal design answers both questions, against each model's own baseline and an equally structured mismatched-state control, scored by a correctness-gated human paired comparison. The exploratory evaluation covered four models across eighteen single-turn, six fixed three-turn and six simulated natural-conversation scenarios. Supplying a synthetic state, raw or translated into a response policy, gave human-preference point estimates above 0.5 for all four models (Claude Sonnet 5, DeepSeek V4 Pro, Gemini 3.6 Flash, GPT-5.6 Terra), with three of eight 95% intervals excluding chance. Specificity was not resolved: no per-model interval excluded chance, and the design cannot yet separate the state values from a generic instruction to adapt. Policy delivery reduced priced cost and output tokens against each model's baseline; these are resource-use changes, not efficiency gains, because correctness non-inferiority was not established. Safety-floor failures were rare (13 of 360 cases), with a non-significant excess under policy delivery. We report the instrument, the frozen results, and the controls a confirmatory study needs.