HealthLoopQA benchmarks AI reasoning on diabetes wearable data
HealthLoopQA: A Context-Aware Question Answering Benchmark for Interpreting Wearable Monitoring Data in Diabetes Care
Machine Learning
Summary
Medical wearables collect lots of health data over time, but understanding this long-term information is hard for patients and doctors. The authors created HealthLoopQA, a test that checks how well AI language models can analyze 30 days of diabetes monitoring data. This test includes tasks like spotting unusual events and predicting future patterns, even when devices misbehave or are hacked. The authors found current AI models struggle with complex long-term patterns and tend to stop trying hard when given lots of information.
What this means in practice
- •For healthcare ai developers: Evaluate and improve AI models’ ability to interpret long-term diabetes wearable data for patient care insights.
- •For medical device engineers: Test wearable device robustness and AI safety response using fault-injected simulated diabetes monitoring scenarios.
Authors
Yuchen Niu, Yanan Ma, Srinivasan Nandakumar, Maolin Chen, Viktor Schlegel, Kexin Wei, Ling Cheng, Anna Bird, Anil Anthony Bharath, Siew-Kei Lam
Abstract
As medical wearables become integrated into daily chronic disease care, effectively interpreting longitudinal monitoring data is essential for patients and clinicians to understand health trends, detect safety-critical events, and make informed decisions. While large language models (LLMs) show promise for transforming this streaming physiological data into personalized health insights, evaluating their reasoning capability and analytical rigor in diverse monitoring tasks remains a fundamental challenge. Existing medical wearable question answering (QA) benchmarks primarily assess short-horizon classification or statistical summaries, largely ignoring the long-term patterns, therapeutic and behavioural contexts, and potential system failures inherent in real-world deployments. To address this, we introduce HealthLoopQA, a comprehensive diagnostic benchmark for evaluating LLM reasoning over continuous diabetes monitoring data. Grounded in a novel taxonomy of eleven atomic reasoning abilities, HealthLoopQA comprises 127 tasks and over 1,500 QA instances spanning process mining, anomaly detection, and prediction over 30-day horizons. To systematically evaluate safety awareness, we complement real-world datasets with a fault-injected simulation testbed modeling diverse device malfunctions and cyber-physical attacks to generate physiologically plausible hazard scenarios. Evaluating state-of-the-art LLMs across prompting and agentic frameworks reveals severe limitations in complex temporal pattern mining. Furthermore, we identify a broader phenomenon of In-context Laziness under long-context prompting, highlighting critical open challenges in deploying LLMs for rigorous long-horizon medical reasoning.