Multimodal dataset improves driver mental state detection across individuals
MultiMental-IAS: A Multimodal Driver Mental State Analysis Dataset for Context-Aware Non-IID Evaluation
Human-Computer InteractionArtificial Intelligence
Summary
Detecting a driver’s mental state is hard because it requires info from the brain, body, behavior, and driving context all at once. The authors created a new dataset by recording these signals from 23 people in a realistic driving simulator while they experienced six different mental states. Their dataset helps test methods that must work well even when drivers are different or unknown to the system. They found that combining driving data with brain and heart signals improved detection accuracy, especially on new drivers.
What this means in practice
- •For automotive safety engineers: Use the dataset to develop and benchmark better driver monitoring systems that adapt to different drivers’ mental states in varying contexts.
- •For wearable device developers: Validate EEG and PPG sensor integrations for monitoring driver mental states alongside vehicle data to improve real-time alerting.
Authors
Baran Can Gül, Nagendra Pal Singh, Anjana Arvind Naik
Abstract
Robust driver-state modelling requires synchronized physiological, behavioural, and contextual signals, yet existing open resources typically target a single cognitive state, omit continuous vehicle context, or do not define evaluation protocols that expose inter-subject heterogeneity. We present Multimental-IAS, a multimodal driver mental-state dataset collected in a cockpit-in-the-loop simulation built on CARLA. It contains vehicle telemetry with scenario-context channels, four-channel EEG and three-channel PPG from a Muse 2 headband, and cabin-facing video from 23 participants, recorded in 148 trials (11.6 hours) under six induced states: Normal, Distracted, Alerted, Stressed, Relaxed, and Drowsy. We document the acquisition pipeline in detail and provide a benchmark with one protocol for all models: chronological within-subject splits, participant-disjoint cross-subject folds, validation-based model selection, and ablations over modalities, scenario-context channels, and signal normalization. Multimodal fusion reaches a macro-F1 of 0.94 for drivers seen during training. For unseen drivers, fusing driving dynamics with per-window normalized EEG/PPG raises macro-F1 from 0.408 (telemetry alone) to 0.485, consistently across all five folds, while the gap between the two protocols quantifies the inter-driver heterogeneity that personalized models must address. Multimental-IAS thus provides a concrete testbed for non-IID generalization, personalization, and federated learning in driver monitoring.