Human pose AI models can be fooled by targeted attacks using keypoint similarity
A Black-Box Adversarial Attack on Human Pose Estimation and Keypoint-Based Action Recognition Models
Computer Vision and Pattern Recognition
Summary
Some computer programs try to figure out human body positions and movements from videos, but these systems can be tricked by small changes that are hard to detect. The researchers created a new way to attack these programs without needing to know their inner details, by focusing on how the body keypoints are measured. Their method lowered the accuracy of pose detection and made it harder for the systems to recognize actions correctly. This shows that these AI tools can be vulnerable and need better protection.
human pose estimationkeypoint-based action recognitionadversarial attackblack-box attackObject Keypoint Similarity (OKS)temporal coherencevideo understandingpose qualitytop-down modelssingle-stage pose estimation
Authors
Kacper Mroczek, Michal Kepski
Abstract
Human pose estimation and keypoint-based action recognition models are increasingly deployed as components of video understanding pipelines, yet their vulnerability to adversarial attacks remains insufficiently studied. Temporally coherent black-box attacks have been previously studied in visual object tracking, where the attack feedback can be defined using bounding-box overlap measures such as Intersection over Union (IoU). However, human pose estimation produces keypoint configurations rather than enclosing boxes, making box-level similarity poorly suited for measuring pose degradation. We propose OKS Attack, a decision-based black-box attack that uses Object Keypoint Similarity (OKS) as the attack feedback signal, directly targeting the spatial structure of human poses rather than their enclosing boxes. Experiments on the Penn Action dataset show that OKS Attack consistently reduces pose quality across evaluated pose estimators, with mean OKS decreases ranging from 0.0802 to 0.1494. In a downstream cross-dataset action-recognition evaluation, the attack reduces accuracy by 6.18 to 13.86 percentage points and outperforms query-matched random-noise perturbations. The attack is effective across both top-down and single-stage pose estimation models. The source code will be made publicly available at https://github.com/KacperM33/OKS_attack