Sensor language action model predicts and explains real world tasks
Sensor-Language-Action Models
Artificial Intelligence
Summary
Sensors collect information about the world, but making decisions about what to do next usually happens separately. The authors created a new type of model that links sensor data, natural language descriptions, and actions all together. This means the model can understand the situation, predict what action to take, and explain why using language. They tested this model in real settings like hospitals and health monitoring, and it worked better than previous methods.
What this means in practice
- •For clinical data teams: Improve patient outcome predictions and decision explanations by integrating diverse sensor data with language and actions in healthcare settings.
- •For health monitoring system developers: Build systems that predict and explain metabolic health actions by connecting sensor readings directly with natural language and behavior predictions.
Authors
Yuekai Xu, Zitao Shuai, Yuzhe Yang
Abstract
Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence. Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over the state-of-the-art. OpenSLA also demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.