Healthcare language models miss varied cultural insights in Indian maternal care
Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts
Computation and Language
Summary
Current AI systems that help with health advice usually check if the info they give is true and safe but don’t see if the advice fits local culture. This matters a lot in Indian pregnancy care, where local traditions shape health choices. The authors created a way to measure if AI advice matches the cultural beliefs of women in urban and semi-urban North India. They found that AI systems often give similar advice to all profiles, unlike real people who vary their opinions based on different backgrounds. Conditioning AI with specific cultural info helps make its advice more accurate and relatable.
What this means in practice
- •For healthcare ai developers: Produce maternal health dialogues that match cultural profiles to improve AI relevance for Indian women
- •For multilingual chatbot designers: Incorporate culturally sensitive conditioning to increase dialogue quality and personalized responses in health bots
Authors
Umaira Izhar, Gunjan Arora, Pushpendra Singh
Abstract
Existing evaluation methods for healthcare LLMs primarily assess factual correctness,safety, and fluency, while providing limited insight into whether generated interactions reflect culturally situated healthcare reasoning. This limitation is particularly important in maternal health, where care decisions are shaped by social and relational norms. We introduce MH-INDIC, a culturally grounded evaluation framework for maternal-health interactions in urban and semi-urban North Indian contexts that operationalises cultural behaviour through ten dimensions of maternal-health reasoning. Using a 26-item survey administered to 102 pregnant and postpartum women from urban and semi-urban North India, we evaluate ten LLMs. We distinguish population level cultural alignment from profile-level behavioural variation. Although several models approximate the human population-level distribution, all evaluated systems exhibit substantially lower variation across demographic and household profiles than the human cohort, revealing a gap between aggregate alignment and profile-conditioned sensitivity. As a downstream application of MH-INDIC, we use the strongest-aligned proprietary and open-source models to generate culturally conditioned maternal-health dialogues under zero-shot, self-conditioned, and human-grounded prompting. Human-grounded conditioning produces stronger profile alignment and dialogue quality ratings, suggesting that measured cultural profiles can improve the cultural grounding of generated interactions