Optimize Surgical Triplet Recognition: A Knowledge-Driven Mixture-of-Experts Solution
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of recognizing actions in robot-assisted surgery by identifying the surgical instrument, action, and target together. They propose a new method called MoeCo that improves this task by separating specialized features, balancing learning for rare categories, and using expert knowledge inspired by surgery to guide the model. Their approach integrates knowledge from language models to make the system more accurate and understandable. Tests on surgical video datasets show their method works better than previous ones.
Surgical action triplet recognitionRobot-assisted surgeryMixture-of-expertsKnowledge-driven learningComponent-level optimizationCategory imbalanceSpatial-temporal featuresLarge language modelsGradient learningCholecT45 dataset
Authors
Yiyi Zhang, Yuchen Yuan, Ying Zheng, Jialun Pei, Jinpeng Li, Zheng Li, Pheng-Ann Heng
Abstract
Surgical action triplet recognition constitutes a critical task in context-aware robot-assisted surgery, facilitating automatic surgical action perception by identifying instrument, verb, target, and their association. However, existing works struggle to analyze such complex surgical scenes due to three main issues: (1) component-level optimization conflicts caused by entangled feature spaces, (2) category-level optimization conflicts arising from severe data imbalance, and (3) lack of domain knowledge guidance that limits model interpretability and robustness. To address these challenges, we propose a Mixture-of-Experts-guided Co-Optimization (\textit{MoeCo}) framework powered by knowledge-driven learning. Within the co-optimization pipeline, to first mitigate component-level conflicts, we introduce a component-tailored adapter that disentangles task-specific features across spatial-temporal regimes, facilitating effective component specialization. Next, we develop a coordinated gradient learning strategy to handle category-level conflicts, which adaptively rebalances positive-negative gradients to enhance the perception of rare categories. Notably, inspired by surgical domain expertise, we introduce a knowledge-driven mixture-of-experts mechanism that dynamically integrates multimodal large language model-guided knowledge via activated experts, thereby enriching the co-optimization pipeline with more expressive and robust representations. Extensive experiments on the public CholecT45 and CholecT50 datasets confirm the effectiveness of the proposed co-optimization pipeline and the superiority of dynamic priors integration via the knowledge-driven mixture-of-experts mechanism.