EmoMed adapts medical advice to emotions using text and images
EmoMed: An Emotionally-Aware Agent for Multimodal Medical Support with Real-Time Information Retrieval
Artificial IntelligenceComputers and SocietyHuman-Computer Interaction
Summary
People often feel worried or confused when asking medical questions. The authors created EmoMed, a computer helper that not only understands your words and pictures but also senses emotions like anxiety or urgency. EmoMed changes how it answers based on how you feel, making responses clearer and more caring without losing accuracy. It double-checks facts using updated medical sources and the internet to stay reliable. Tests show that people find its help more empathetic and easy to understand while trusting the information.
multimodal processingemotion detectionmedical consultation agentreal-time information retrievalclinical accuracyLLM evaluationfact-checkinguser studyempathy in AImedical knowledge base
Authors
Ivan Nasonov, Nikita Glazkov, Ivan Makovetskiy, Mikhail Mozikov, Daniil Sukhorukov, Andrey Savchenko, Ilya Makarov
Abstract
We present EmoMed - a multimodal medical consultation agent that adapts its responses based on users' emotional states while maintaining clinical accuracy. The system processes text and medical images, detects affect indicators (anxiety, confusion, urgency) from user input, and adjusts response tone, structure, and detail level accordingly. To ensure factual reliability, the agent grounds clinical information through a dual retrieval mechanism: web-based fact-checking and an API-connected, continuously updated medical knowledge base. We evaluate our approach across seven state-of-the-art language models (GPT-4/5, Qwen3, Llama 4, Gemini 2.5, Grok4, Claude3) using comprehensive metrics including LLM-as-judge assessments, MedQA style accuracy tests, BERT Score, safety/helpfulness ratings, and multimodal medical benchmarks. The results demonstrate that emotionally adaptive responses consistently outperform neutral baseline across evaluation dimensions, without compromising clinical accuracy. A controlled user study validated these findings, with participants reporting improved perceived empathy and communication clarity, while maintaining trust in factual accuracy. Source code: https://github.com/NasonovIvan/EmoMed-Agent