TimeCap creates scalable captions for health wearable data
TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data
Machine LearningComputation and Language
Summary
Many health devices collect complex data over time, but understanding this information is hard because it comes from many sources and types. The authors introduce TimeCap, a tool that automatically creates detailed, reliable descriptions of this data without using unpredictable AI language models. They used TimeCap to build MHCap, a huge open collection of wearable health data combined with these descriptions. Training time-series language models on MHCap improved predictions about health outcomes compared to using labels alone.
What this means in practice
- •For wearable device developers: Generate reproducible, detailed captions for multimodal wearable sensor data to improve device data interpretation and model training.
- •For healthcare data teams: Integrate large, open captioned time-series datasets like MHCap to enhance machine learning models for predicting patient health outcomes.
Authors
Max Rosenblattl, Milana Gurbanova, Karl Aurel Deck, Narayan Schuetz, Thomas Kaar, Manuel Milling, Nicolas Zumarraga, Ning Wang, Kelly W. Zhang, Anders Johnson, Robert Jakob, Björn W. Schuller, Elgar Fleisch, Daniel Seung Kim, Allan Lawrie, Patrick Langer, Maxwell A. Xu, Euan Ashley, Paul Schmiedmayer
Abstract
Time-series data generated continuously across industrial, environmental, and especially health domains produce multimodal signals whose scale and heterogeneity make their processing and interpretation particularly challenging. Time-Series Language Models (TSLMs) promise scalable interpretation of these signals, yet remain bottlenecked by a fundamental lack of text-annotated time series pairs that are signal-grounded, reproducible, and extensible across modalities and schemas. Existing paired corpora are either closed or limited in scale and modality coverage. Annotation with large language models could scale them up, but its outputs cannot be regenerated exactly, which prevents others from re-deriving, extending, or auditing the resulting corpus. To address this, we release TimeCap, an open deterministic captioning pipeline with three reusable extractor tiers and a cross-channel synthesis layer that operates directly on the signal with no language model in the loop, extensible to new schemas by writing channel definitions alone. This positions TimeCap as a reproducible mechanism for future dataset generation rather than a one-off release. Applying TimeCap to OpenMHC, the MyHeart Counts wearable dataset, yields MHCap, the largest open caption layer for multimodal wearable time-series data. MHCap spans 19 Apple HealthKit channels and 7,005,763 captions over 908,944 person-days. To showcase the impact of the corpus, we train OpenTSLM, a time-series language model, on MHCap and benchmark on health-outcome prediction. Caption-augmented training (OpenTSLM-Cap) outperforms a label-only baseline on 29 of 32 per-user binary evaluations across 8 tasks.