Toward Trustworthy Large Language Model Agents in Healthcare
2026-07-06 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created CareConnect, a chatbot designed to help schedule healthcare appointments more efficiently by automating tasks like booking, changing, or canceling visits. It uses advanced language technology combined with strict safety rules to avoid giving medical advice or handling emergencies unsafely. Their tests showed the system works well, completing most tasks quickly and safely while reducing costs compared to human schedulers. The authors also made their code publicly available for others to use and improve.
healthcare schedulinglarge language modelsfunction callingretrieval-augmented generationconversational agentssafety guardrailstask automationcost efficiencyemergency detection
Authors
Hadi Hasan, Safaa Salman, Adam Tai Abou Dargham, Ammar Mohanna, Ali Chehab
Abstract
Healthcare appointment scheduling remains a persistent operational bottleneck, driven by manual coordination, fragmented legacy systems, and high administrative overhead. These inefficiencies constrain provider availability and degrade patient access to care. This paper presents CareConnect, a safety-first conversational agent for healthcare logistics automation that leverages large language model (LLM) function calling, retrieval-augmented generation (RAG), and layered deterministic safety guardrails. The system orchestrates eight domain-specific tools to support appointment booking, modification, cancellation, and facility information retrieval, while enforcing strict scope constraints that prohibit medical advice or diagnosis. Safety-critical situations are handled through deterministic short-circuit mechanisms for emergency detection and medical intent refusal. We evaluate CareConnect on a comprehensive benchmark of 680 task-oriented scenarios spanning end-to-end workflows, multi-turn interactions, and edge cases. Experimental results demonstrate a 91.8% task completion rate with a median per-request latency of 2.2 seconds, 96.0% safety compliance on the dedicated safety-critical evaluation subset, and an average operational cost of $0.0324 per appointment, yielding a significant cost reduction compared to manual human scheduling. These findings show that carefully scoped and rigorously safeguarded LLM-based agents can reliably automate complex healthcare operational workflows while maintaining safety guarantees and achieving substantial cost efficiency. The source code and system implementation are publicly available at https://github.com/Hadi-Hsn/CareConnect.