ReMiX-MAE: Learning Missing-Channel Cross-Modal Representations from RGB-Only Clinical Facial Videos for Sympathetic-Mediated Pain Assessment
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created a new method called ReMiX-MAE that learns from videos showing faces in three types of images: regular color (RGB), heat (thermal), and 3D shape (depth). Their method trains the computer to handle missing types of images, so it can still work using just regular color video, which is easier to collect. They also made a new dataset with videos of people before and after pain treatment to test their method. Their approach performed better than simpler methods, especially when predicting different levels of pain, and worked well even when there was little training data.
automated pain assessmentmasked autoencodermultimodal learningRGB videothermal imagingdepth sensingself-supervised learningclinical datasettransfer learningfacial representation
Authors
Nan Bi, Taoyue Wang, Lijun Yin, Vandana Sharma
Abstract
Automated pain assessment in real clinics is limited by scarce clinically grounded facial video data with weak labels (often sequence-level self-report) and by the fact that pain cues can be subtle or near-neutral in RGB, while thermal and depth signals are informative yet impractical to deploy routinely. To address these challenges, we propose ReMiX-MAE (Reconstructing Missing Channel Cross-Modal Masked Autoencoder), a self-supervised multimodal masked pretraining framework that learns transferable facial representations from synchronized RGB, thermal, and depth videos and explicitly trains robustness to missing modalities, enabling RGB-only deployment. To fill the gap of clinically grounded facial pain data with video-level self-report and longitudinal treatment trajectories, we collect the Sympathetic Mediated Pain (SMP) dataset with paired pre- and post-recordings across multiple visits. Under RGB-only deployment, we evaluate ReMiX-MAE using both direct feature extraction and pseudo-multimodal features decoded from RGB. ReMiX-MAE consistently outperforms an RGB-only masked autoencoder baseline on SMP, with pseudo-multimodal features providing additional gains in the challenging five-class setting. Across external datasets, ReMiX-MAE further shows more robust and label-efficient transfer than RGB-only baselines, highlighting its advantage in data-limited clinical settings.