Adaptive sensor weighting improves human activity recognition accuracy and cost
Federated Multi-Modal Human Activity Recognition using Multi-Agent Reinforcement Learning
Artificial Intelligence
Summary
Human activity recognition helps track what people do using sensors, which is important for healthcare and rehabilitation. The authors designed a new way to combine information from different sensors by giving more attention to the best sensors for each situation and ignoring less useful or costly ones. They used a method where separate agents learn how to weigh sensor data dynamically, and extended this approach to allow many devices to learn together without sharing raw data, improving privacy. Their approach showed better accuracy and stability compared to older methods, while also reducing sensor use costs.
What this means in practice
- •For healthcare technology teams: Improve activity monitoring systems by dynamically adjusting sensor input importance to increase accuracy and reduce cost in patient rehabilitation devices.
- •For mobile device manufacturers: Develop wearable devices that selectively use sensors to balance performance and battery life by applying adaptive sensor weighting learned through federated multi-agent methods.$Commercial implications: Enables manufacturing of more efficient and accurate wearable health monitors that reduce sensor usage and prolong battery life, appealing to consumer markets.
Authors
Debasmita Dey, Tanmay Sen, Himel Mallick
Abstract
Human Activity Recognition (HAR) from heterogeneous wearable sensors is fundamental to the Internet of Health Things (IoHT), supporting rehabilitation, elderly care, and smart healthcare. Existing multimodal fusion methods often assign fixed equal weights to sensor streams, overlooking differences in modality importance, acquisition cost, and sensor quality, which can vary due to movement, incorrect placement, or temporary blockage. We propose an adaptive and cost-aware multimodal HAR framework based on multi-agent reinforcement learning for centralized HAR and extend it to federated learning as FedMHAR. In the centralized setting, multimodal fusion is formulated as a cooperative Multi-Agent Reinforcement Learning (MARL) problem, where each sensing modality is assigned a PPO-based agent that learns per-sample fusion weights, enabling the model to emphasize informative modalities while down-weighting costly sensors when cheaper alternatives provide sufficient information. In the federated setting, we introduce BiFL-PPO, a bidirectional federated optimization strategy in which a server-side PPO policy learns client-specific trust weights and feeds them back to adapt local learning rates and proximal regularization. Unlike round-level optimization, BiFL-PPO uses dense batch-level rewards for more frequent feedback and stable training under heterogeneous client data. Evaluation on the MEx Rehabilitation and UTD Multimodal Human Action datasets shows that the centralized framework achieves 87.30% and 94.98% accuracy, respectively, outperforming conventional fusion methods and state-of-the-art HAR models. FedMHAR achieves 79.74% and 77.49% in the federated setting, consistently surpassing FedAvg, FedProx, FedBN, FedNova, and AdaFedProx, while providing more stable performance and reducing sensor acquisition cost.