Vision language models struggle with detailed physiological readings

Why Do VLMs Fail at Remote Physiological Measurement? Rethinking with Physiological Grounding and Numerical Alignment

Computer Vision and Pattern Recognition

Summary

Measuring heart rate and other body signals from videos usually needs special models, but big vision-language models (VLMs) that understand images and text together have trouble with this. The authors found two main reasons: these models miss small physiological details in pictures, and their way of predicting words doesn’t fit well with measuring ongoing numbers like heart rates. They created a new method called VITAL that keeps normal image understanding but adds a separate system to catch body signals and better handle number predictions. Tests show VITAL works better on multiple heart rate datasets while still understanding images normally.

What this means in practice

Authors

Xiao Yang, Zekun Zhang, Yuxuan Fan, Duoxun Tang, Ziyang Liu, Dengbo He, Kaishun Wu, Jiyao Wang

Abstract

Video-based remote physiological measurement (RPM) has broad applications, while Vision-Language Models (VLMs) provide a promising opportunity to enable end-to-end physiological monitoring for future human-aware intelligent systems. However, existing RPM methods rely on task-specific models and cannot be seamlessly integrated with the multimodal understanding and interaction capabilities of VLMs. We revisit RPM with VLMs and provide empirical evidence for two major bottlenecks: generic visual semantic representations fail to preserve fine-grained physiological cues; while discrete token prediction in autoregressive generation is inherently misaligned with continuous numerical measurement. To address these challenges, we propose VITAL (Visual-physiological Token Alignment for Language Models), a visual-physiological token alignment framework for RPM and preserving general vision tasks. VITAL retains the original visual pathway for semantic understanding and introduces an independent physiological pathway to jointly model semantic and physiological information. Moreover, we propose a structured numerical alignment approach to improve the reliability of generative heart rate estimation by enhancing numerical relationship modeling. Extensive experiments show that VITAL improves cross-domain estimation performance on multiple rPPG datasets while preserving the general visual understanding capability of VLMs, demonstrating its effectiveness as a unified visual-physiological model. Code will be released.
View PDFOpen arXiv