A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedback
2026-05-25 • Computation and Language
Computation and LanguageArtificial IntelligenceMultiagent Systems
AI summaryⓘ
The authors studied how surgeons give verbal feedback to trainees during live surgeries and found it hard to measure how good and effective that feedback is. They created a new two-step method using large language models (LLMs) to identify important qualities of feedback, like being clear or urgent. Their approach can automatically score feedback and does better than older methods at predicting if the feedback helps trainees improve. This work could help make surgical training communication easier to evaluate and improve.
verbal feedbackresident traineeoperating roomlarge language modelmulti-agent promptingfeedback qualitysurgical trainingbehavioral adjustmentcommunication assessmentAI scoring
Authors
Rafal Kocielnik, J. Everett Knudsen, Steven Y. Cen, Jasmine Lin, Cherine H. Yang, Atharva Deo, Ujjwal Pasupulety, Peter Wager, Anima Anandkumar, Andrew J. Hung
Abstract
Verbal feedback delivered by attending surgeons in the operating room plays a critical formative role in resident trainee skill acquisition. Yet, assessing the quality of trainer feedback and its effectiveness in influencing trainee behavior during live surgery remains a challenge. Prior studies assessed feedback content relying on extensive manual annotation by expert human raters and focused on developing broad taxonomies that overlook the qualitative aspects of feedback delivery such as clarity or urgency. Limited existing automated methods, including keyword analysis and topic modeling, also fail to capture these nuanced aspects. We introduce a two-stage LLM-based framework that discovers interpretable feedback quality criteria grounded in the context of surgical training. Our method uses multi-agent prompting and surgical domain knowledge injection to discover a small set of human interpretable scoring criteria (e.g., Encouraging, Urgent, Clear). These criteria are then used to automatically score live surgical feedback via an LLM-as-a-judge approach. Evaluation on 4.2k trainer feedback instances demonstrates that our AI-discovered criteria outperform prior content-based frameworks in predicting feedback effectiveness, including observed trainee behavioral adjustments and trainer approval. This work advances scalable, human-aligned assessment of communication quality in the operating room and provides a foundation for improving surgical teaching practices.