Large language models improve safety in Japanese stroke care conversations
Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models
Computation and Language
Summary
Stroke care requires careful questions and decisions to avoid life-threatening mistakes. This study tested how well large language models (LLMs) can act like doctors in Japanese conversations about stroke cases. Some models did very well and avoided dangerous errors, showing they could help in safe stroke care. The research also created a special test made by neurosurgeons to measure these skills. More work is needed to see how these models do with real patients.
What this means in practice
- •For medical device developers: Develop conversational AI tools that safely assist physicians in Japanese stroke diagnosis and management following neurosurgeon-designed safety benchmarks.$Commercial implications: Enables creation of AI-powered stroke care assistants certified for safety, addressing a clinical need in Japanese healthcare.
- •For clinical quality assurance teams: Use the Japanese Stroke LLM Evaluation benchmark to test and monitor clinical AI systems’ safety and performance in simulated stroke care scenarios.
Authors
Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto, Hirofumi Hirano, Ryosuke Hanaya
Abstract
Background: Large language models (LLMs) have achieved physician-comparable performance on multiple-choice medical knowledge examinations, but their capabilities in clinical history taking, urgency assessment, and safety remain insufficiently evaluated. We proposed Japanese Stroke LLM Evaluation, a multi-turn conversational benchmark for stroke care in Japanese, and evaluated LLM performance and safety under practice-oriented conditions. Methods: We created 10 stroke and related-condition cases and evaluated LLMs in multi-turn Japanese conversations. The LLM acted as physician, while a board-certified neurosurgeon acted as simulated patient and evaluator. Each case comprised history-taking and action phases scored using pre-specified criteria. Errors that could directly threaten life were defined as critical mistakes. The safety threshold was at least 80% overall with zero critical mistakes. Eighteen models were evaluated in October 2025 and June 2026. Results: Claude Fable 5 achieved the highest score (87.4%) with zero critical mistakes, followed by Claude Opus 4.7 (80.3%) and GLM-5.2 (75.6%). Two leaders met the safety threshold. Eleven models made 17 critical mistakes, including failure to confirm laboratory results or blood glucose before t-PA, surgery before airway stabilization, omission of cervical vascular evaluation, and t-PA outside its indication. History-taking question count correlated with history-taking score (r = 0.648, p = 0.007). Conclusions: Japanese Stroke LLM Evaluation provides a benchmark for LLM performance under practice-oriented conditions, including a cap on history-taking questions. Cases and evaluations were created by neurosurgical specialists rather than using an LLM-as-judge approach. Performance improved across cloud-based and on-premise models in 2026, with some exceeding the safety threshold. Further evaluation using real-world cases is required.