Hierarchical explainability framework reveals timing modality and features in ai predictions
T-MoXAI: A Hierarchical Explainability Framework for Temporal Multimodal Data
Machine Learning
Summary
Artificial intelligence can help predict important things like medical outcomes or crop yields using data collected over time from different sources like images and measurements. But these AI models are often hard to understand, which makes people hesitant to trust them. The authors present T-MoXAI, a tool that explains exactly when in time the model pays attention, which type of data it trusts most at those times, and what specific details it uses to make decisions. This helps users see not just what the AI predicts but why, making it easier to trust and improve.
What this means in practice
- •For clinical data teams: Support fertility specialists by showing when and how AI uses ultrasound and clinical data to predict IVF outcomes, making AI-assisted decisions more transparent.
- •For agricultural analytics providers: Provide crop yield forecasts with explanations revealing which time points and data types drive predictions from satellite images and phenotypic traits.
Authors
Ali Inha, Mo Vali, Saaliha Vali, Pietro Liò, Meen-Yau Thum
Abstract
Artificial Intelligence (AI) models for temporal multimodal data have potential in healthcare and agriculture, but their opacity can limit trust and adoption. We introduce T-MoXAI (Temporal Multimodal eXplainable AI), a hierarchical framework explaining (1) when timepoints influence predictions, using temporal Shapley values; (2) which modalities contribute at those moments, using attention analysis; and (3) what features or image regions drive decisions, using gradient based attribution. A transformer based architecture handles irregular temporal sequences and heterogeneous data, generating all three explanation levels in under one second for interactive decision support. We evaluate the framework on two real world tasks: predicting IVF treatment outcomes from ultrasound sequences and clinical measurements (AUC 0.660 despite significant class imbalance), and forecasting wheat yield from temporal RGB imagery and phenotypic traits ($R^2$ 0.265 amid substantial environmental variability). Ablation studies indicate that temporal modelling is critical in both domains: removing it reduces performance to the equivalent of random guessing. Temporal ROAR experiments provide evidence that the explanations reflect the model's reasoning process. With a unified, domain agnostic architecture and open source implementation, T-MoXAI provides a baseline for temporal multimodal XAI, addressing fragmentation in the field and supporting applications where understanding decisions is as important as predictive accuracy.