MemoCare provides automated multimodal cognitive screening on mobile devices
MemoCare: An Interactive Multimodal Mobile System for Automated Cognitive Screening
MultimediaComputer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Checking how well someone’s brain is working can take a lot of time and expert skills. The authors created MemoCare, a phone app that asks questions, listens to answers, sees where you are, watches your screen touches, and checks drawings to see how your brain is doing. It works with English and Vietnamese and scores all tasks automatically using special computer programs and models. Medical experts tested it and gave mostly positive ratings, showing it could help make brain tests easier to give and score.
What this means in practice
- •For mobile app developers: Create health apps that perform automated cognitive tests using speech, location, touch, and drawing data with multi-language support.
- •For clinical operations teams: Use a unified mobile system to speed up cognitive screen testing while automating scoring and supporting English and Vietnamese patients.
- •For digital health startups: Develop commercial cognitive assessment services by integrating multimodal scoring and language processing to reach diverse populations efficiently.$Commercial implications: Enables selling smartphone-based cognitive screening tools with automated scoring and multilingual workflows to healthcare providers.
Authors
Duy-Cat Can, Mau Minh Phuc Le, Tuan-Khoa Hoang, Hai-Dang Nguyen, Trung-Hieu Do, Dang Minh Ly, Minh-Duc Nguyen, Nghia TT Hoang, Linh-Trung Nguyen, Huy-Hieu Pham, Huong Ha, Binh T. Nguyen, Oliver Y. Chén
Abstract
MemoCare is an interactive mobile system for automated multimodal cognitive screening. A React Native application combines spoken responses, temporal and spatial orientation, touchscreen actions, and visuoconstruction in complete English and Vietnamese workflows. Speech is transcribed by Google Speech-to-Text and scored locally with deterministic task-specific natural language processing rules; GPS coordinates are resolved by the MemoCare spatial module before answer matching; touch tasks are scored from interaction events; and the drawing task uses a three-model convolutional neural network consensus with separate visual interpretation. Software tests pass 151/151 predefined cases across speech/language, spatial-answer, and touch-interaction scoring, while spatial regression passes 48/48 four-country coordinate-resolution cases. For the drawing module, validation-selected ShuffleNetV2 x1.5 achieved 91.33% mean balanced accuracy and 78.87% exact three-criterion accuracy on a locked 71-image test set. Four clinician co-authors additionally inspected the end-to-end workflow, yielding a pooled median rating of 4/5 across eight criteria, with item-level medians ranging from 3 to 4.5. At MMM, attendees can directly try a shortened multimodal screening workflow and inspect automatic item-level and total scoring.