AdaptaFace boosts facial gesture recognition for people with motor impairments
AdaptaFace: Adversarial Assistance for Inclusive Facial Gesture Interaction
Human-Computer Interaction
Summary
Facial gestures can help people who have difficulty moving their bodies communicate or control devices, but computers often struggle to understand their unique facial movements. The authors created AdaptaFace, a smart software layer that modifies a person's facial gesture inputs so existing recognizers can understand them better without retraining. It improved completion rates and reduced effort in a user study while allowing users to add new gestures. This approach uses special small changes to the input to make recognition more accurate and inclusive.
What this means in practice
- •For assistive technology developers: Improve accessibility tools by enabling personalized facial gesture recognition without retraining models for each user.
- •For mobile device engineers: Integrate adaptive facial gesture input layers to enhance user interaction for motor-impaired users on existing AU-based recognition systems.
Authors
Siyu Zhang, Yelu Gu, Xiaoyun Xu, Kirsten Cater, Oussama Metatla
Abstract
Facial gestures can enable access for people with motor impairments (PwM), yet recognizers may not reliably interpret individual facial movement patterns. Existing approaches either constrain agency by requiring predefined templates or impose repeated calibration, data collection, and computation through user-specific retraining. We present AdaptaFace, an input-level compatibility layer using bounded, proxy-dependent adversarial perturbations to translate user-compatible gestures into inputs for an unchanged facial action unit (AU)-based recognizer. Designed around six personalization principles informed by interviews with 12 PwM, AdaptaFace supports both recognition enhancement and gesture vocabulary extension. In a within-subject study with 16 participants, AdaptaFace increased task completion from 59.7% to 90.5% and reduced workload from 5.61 to 4.21. Offline validation showed strong steering with aligned perturbation generation and evaluation but weaker cross-recognizer transfer, defining the approach's boundary. These findings frame adversarial assistance as input-side accessibility mediation for personalized facial-gesture interaction.