New protocol reveals flaws in human object interaction detector scores
Auditing Semantic and Relational Reliability in Open-Vocabulary Human-Object Interaction Evaluation
Computer Vision and Pattern Recognition
Summary
Evaluating computers that describe how people interact with objects is tricky because they use open-ended language. The authors created a way to check if these evaluations unfairly reject correct answers or mistakenly accept wrong ones. Their method tests predictions from multiple angles to separate true understanding from simple text matching. When tested on different systems, they found that evaluation scores could be misleading, recommending more comprehensive reporting. This helps developers better understand how well these detectors really grasp interactions.
What this means in practice
- •For computer vision engineers: Improve evaluation of interaction detection systems by applying a multi-view protocol that better identifies semantic and relational correctness.
- •For natural language processing developers: Use detailed scoring methods to reduce false credit for interaction predictions that rely on lexical matches but lack correct meaning.
Authors
S M Asif Hossain, Ruksat Khan Shayoni, Md Kishor Morol
Abstract
Open-vocabulary human-object interaction detectors describe interactions in free text, and this creates a problem for evaluation. Category scoring rejects valid paraphrases, while semantic similarity scoring can reward predictions that name the wrong action or reverse the roles of the participants. We present a diagnostic protocol that holds every prediction fixed and scores it under five views, namely official category evaluation, an identity control, lexical normalization, graded semantic credit, and strict propositional equivalence. Comparing these views separates the effect of the matching engine from the effect of semantic credit. The protocol pairs controlled transformations that preserve or break meaning with a blinded audit of natural predictions, and we derive consistency checks that link mean credit, invalid acceptance, and transformation counts. Applied to six systems on HICO-DET, the views disagree on which system leads. GEN-VLKT leads under official evaluation, SL-HOI leads under SHOE, and GEN-VLKT leads again under strict equivalence. Semantic gains range from 1.6 to 20.3 mAP, and AgentHOI keeps less than half of its gain once equivalence is enforced. We recommend that semantic scores be reported together with relational validity and paired uncertainty, so that recovered credit is not read as better interaction understanding.