Black-box attack lowers accuracy of human pose and action models
A Black-Box Adversarial Attack on Human Pose Estimation and Keypoint-Based Action Recognition Models
Computer Vision and Pattern Recognition
Summary
Human pose estimation and action recognition models try to understand body movements in videos, but this paper shows they can be fooled by special subtle changes. Instead of using simple box comparisons, the authors created a new way to attack these models by confusing how key body points are detected. Their method lowers the accuracy of pose detection and action recognition across many models. This shows these systems are vulnerable to attacks that work without knowing model details.
What this means in practice
- •For video surveillance teams: Evaluate and improve robustness of human pose based tracking against realistic adversarial attacks that target keypoints instead of bounding boxes.
- •For security engineers: Develop defenses for pose estimation systems vulnerable to black-box attacks that work without requiring model details or gradients.
Authors
Kacper Mroczek, Michal Kepski
Abstract
Human pose estimation and keypoint-based action recognition models are increasingly deployed as components of video understanding pipelines, yet their vulnerability to adversarial attacks remains insufficiently studied. Temporally coherent black-box attacks have been previously studied in visual object tracking, where the attack feedback can be defined using bounding-box overlap measures such as Intersection over Union (IoU). However, human pose estimation produces keypoint configurations rather than enclosing boxes, making box-level similarity poorly suited for measuring pose degradation. We propose OKS Attack, a decision-based black-box attack that uses Object Keypoint Similarity (OKS) as the attack feedback signal, directly targeting the spatial structure of human poses rather than their enclosing boxes. Experiments on the Penn Action dataset show that OKS Attack consistently reduces pose quality across evaluated pose estimators, with mean OKS decreases ranging from 0.0802 to 0.1494. In a downstream cross-dataset action-recognition evaluation, the attack reduces accuracy by 6.18 to 13.86 percentage points and outperforms query-matched random-noise perturbations. The attack is effective across both top-down and single-stage pose estimation models. The source code will be made publicly available at https://github.com/KacperM33/OKS_attack