Millimeter wave radar enables robust in air handwriting recognition
mmPencil: Toward Writing-Style-Independent In-Air Handwriting Recognition via mmWave Radar and Large Vision-Language Model
Human-Computer InteractionComputer Vision and Pattern Recognition
Summary
Recognizing handwriting made in the air is tricky because people write differently and in many ways. The authors created a system called mmPencil that uses special radar to track hand movements in 3D and turns these movements into images. Then, it uses a large vision-language model to understand the handwriting regardless of how or where the writing happens. Their method works well even when different people write differently or under different conditions, and it can recognize many words with high accuracy.
What this means in practice
- •For mobile device manufacturers: Integrate in-air handwriting recognition for input without touchscreens using mmWave radar and vision-language models.$Commercial implications: Enables new hands-free input features for smartphones and wearable devices, creating market differentiation.
- •For assistive technology developers: Create communication aids that allow users to write words in the air accurately across varied handwriting styles and conditions.
Authors
Yifan Guo, Zhu Wang, Qian Qin, Yangqian Lei, Qiwen Gan, Zhuo Sun, Chao Chen, Bin Guo, Zhiwen Yu
Abstract
In-air handwriting recognition based on radio frequency (RF) signals offers a promising solution for human-computer interaction, yet existing methods are often limited by predefined writing planes, constrained writing styles, and small recognition vocabularies. To address these limitations, we propose m^2VLMs, a modality mapping architecture that bridges millimeter-wave (mmWave) radar sensing and large vision-language models (VLMs), and develop mmPencil, a 3D in-air handwriting word recognition system. mmPencil reconstructs 3D handwriting trajectories from mmWave signals using a multi-stage spatial trajectory reconstruction algorithm and maps them into visual representations through a spatial-to-visual mapping algorithm, enabling VLMs to recognize handwriting while reducing sensitivity to writing styles and environmental factors. We collect and release a 3D mmWave handwriting dataset containing 7,664 samples of 200 words from 12 users across 22 writing scenarios. Extensive experiments demonstrate that mmPencil achieves robust word recognition under variations in writing position, range, angle, speed, size, direction, and user movement. It achieves 97.60% accuracy on 200 word classes for seen users and 92.50% zero-shot accuracy on 50 word classes for unseen users, outperforming state-of-the-art baselines.