SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created a test called SafeGesture to see how well vision-language models understand hand gestures in important safety situations. They found that while some models are very good at recognizing gestures, they struggle to correctly decide what safety action to take based on the situation. Most models are not good at admitting uncertainty, and simply guessing the most common safety action without looking at the image often beats the models. Adding images helps performance more than just giving the exact gesture as text. This suggests the harder part for these models is thinking about the safety context, not just seeing the gestures.
vision-language modelshand gesture recognitionsafety actionsscenario-conditioned reasoningbenchmarkQwen2.5-VLGPT-4ooperation scenarioslabel biasmacro-F1 score
Authors
Taegang Kim, Saleh Afroogh, Junfeng Jiao
Abstract
Open-weight and frontier vision-language models (VLMs) perform well on general image understanding, but their ability to interpret fine-grained hand gestures in safety-critical operational contexts remains largely unexamined. We introduce SafeGesture, a benchmark that evaluates whether a model can infer scenario-appropriate safety actions from hand gestures. It pairs six HaGRID gestures with eight operational scenarios for 4,800 items and evaluates Qwen2.5-VL-7B, LLaVA-NeXT-7B, InternVL2-8B, Phi-3.5-Vision, and GPT-4o. Results reveal a perception-reasoning decoupling: GPT-4o achieves 98.4% gesture accuracy but 53.3% safety accuracy, while Qwen2.5-VL reaches 84.9% and 39.5%, yielding gaps of 45.0 and 45.4 percentage points. Four of five models rarely or never use the uncertainty label, and failure directions differ substantially across models. Accuracy also obscures label bias: a scenario-majority policy with no visual input reaches 58.3%, above every evaluated model, while only GPT-4o exceeds this prior under macro-F1. Visual input improves safety accuracy by 11.2 to 30.2 percentage points, but providing the ground-truth gesture as text improves performance by only 0.4 to 3.2 points, and no model exceeds 56.2%. These results indicate that the main bottleneck is scenario-conditioned safety reasoning rather than gesture recognition.