Papers for

surgical robotics teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Automated endoscope control responds to surgeons using language

Language-Conditioned Behavioral Cloning for Autonomous Endoscope Guidance

Abstract: Automated endoscope guidance reduces the need for continuous manual camera control during minimally invasive surgery. However, the desired camera motion depends on the surgeon's immediate intentions and cannot always be inferred from the visual operative scene alone. We propose to automate the task using language-conditioned behavioral cloning; we define 6 clinically relevant guidance behaviors and collect 235 subtask demonstrations. In preliminary phantom experiments, our policy achieves 100% success on focus, overview, and stop, 80% on search-tip and follow-tip behaviors, and 50% on horizon leveling. These results demonstrate the feasibility of learning heterogeneous surgeon-requested camera behaviors with a single visuomotor policy, while the observed failure modes motivate future work on adding temporal context for more demanding guidance tasks.

Fri 30 OctRobotics
The gist
During surgeries with small cuts, doctors use tiny cameras called endoscopes to see inside the body. Controlling these cameras is hard because the doctor’s wishes about where to look change all the time, and it’s not always clear just from the video. The authors taught a computer to move the camera by listening to different spoken commands, like focusing or following a tool tip. Their tests show the computer can successfully follow many commands, proving this kind of language-guided camera control is possible.
Open → 2610.41978v1

Robot takes surgical tool from surgeon without real training

SimRetract: Zero-Shot Sim-to-Real Robotic Assistance for Retractor Take-Over in Open Surgery

Abstract: We demonstrate the first zero-shot sim-to-real transfer for a surgeon-robot collaboration task in open surgery: retractor take-over. A robot arm takes a retractor from the surgeon's hand and preserves retraction at handoff, using a policy trained only on simulated demonstrations, without human demonstrations or real-world fine-tuning. Imitation learning from real demonstrations is limited by collection and labeling costs, and by performance degradation when the environment changes. Our framework, SimRetract, generates simulated demonstrations designed for transfer: the policy observes masked instrument images and grasp geometry, learns to recover from off-nominal states, and acts with margin for execution drift. Deployed zero-shot, it grasps the retractor in 90% of 50 surgeon-defined phantom presentations and preserves retraction at handoff in 78%. Results on unseen retractors suggest potential for generalization to instruments with similar grasp geometry. Across ten policy configurations and 1,200 simulated and real trials, we localize the sim-to-real gap to components: in simulation, grasp rates across ablations span 80-96%, while on the real robot removing the instrument mask alone drops performance to 22% grasped and 12% held. Transfer-oriented design thus makes zero-shot sim-to-real transfer feasible for a surgical assistance task without human demonstrations. Code is available here: https://anonymous.4open.science/r/simretract-58D2.

Wed 28 OctRobotics
The gist
Taking tools from a surgeon during open surgery is tricky because robots need careful training. This paper shows a robot can learn to take a retractor—a tool that holds open a surgical area—using only computer simulations and then perform the task in real life without extra training. The robot watches masked instrument images and understands how to grasp the tool, even when things aren’t perfect. The authors’ method works well on new retractors, suggesting it could handle similar tools too.
Open → 2610.38234v1

Autonomous robots reconstruct and manipulate surgical threads live

Live Thread Reconstruction for Autonomous Manipulation

Abstract: Autonomous robotic surgery has drawn increased attention in recent years due to its potential to relieve surgeons of time-consuming, mentally straining, and tedious tasks, particularly during prolonged operations. One suitable task for automation is suturing, which involves tracking the thread as it deforms during needle retrieval, de-tangling, knot tightening, and other tasks. Unlike rigid-object manipulation, deformable thread manipulation has no fixed pose to track and effectively infinite degrees of freedom, making its configuration difficult to predict and requiring the thread's state to be re-estimated rather than assumed. At the same time, thread manipulations involve sequential motions that depend on past and current thread contact points, which must be accurately and continuously tracked. This paper presents an image-to-manipulation pipeline that concurrently generates live thread reconstruction while conducting safe and optimized sequential manipulations to achieve long-horizon tasks.

Thu 22 OctRobotics
The gist
Suturing stitches during surgery is a tricky, time-consuming task because the thread moves and twists in complex ways. The authors present a method that uses images to continuously track and rebuild the shape of the thread in real time. This helps a robot perform the suturing steps safely and efficiently, even as the thread bends, knots, or gets tangled. Their system improves the robot’s ability to handle these flexible threads during long and complex tasks.
Open → 2610.30694v1

SurgiCLAW improves precision skill control for brain surgery robots

SurgiCLAW: An Interpretable Atomic-Skill Agent for Precision Robotic Micromanipulation

Abstract: Recent multimodal large language models (MLLMs) and vision--language--action models have provided robots with new capabilities for long-horizon task understanding and planning. In surgical robotics, however, end-to-end prediction of low-level actions remains limited by insufficient precision, opaque decisions, and scarce data. This paper presents SurgiCLAW, a hierarchical skill-based framework for precision surgical manipulation. SurgiCLAW confines an MLLM to intraoperative visual understanding, structured explanation, and atomic surgical-skill selection, while delegating physical execution to verifiable deterministic control programs. For brain--machine-interface implantation, a representative high-precision surgical task, we construct a state-to-skill invocation dataset and expand it through visual and state perturbations. The model first learns structured skill invocation through supervised fine-tuning and is then optimized using three rewards for action correctness, operation prerequisites, and format validity, improving action selection, operational safety, and output parseability. We evaluate subtask success, offline model comparison, real-robot performance, and ablations. On a fixed test set, SurgiCLAW achieves 94.6% skill-selection accuracy and 93.0% macro-averaged F1 (Macro-F1). Across 50 closed-loop real-robot implantation trials, it achieves an 88.0% task success rate.

Sun 11 OctRoboticsArtificial Intelligence
The gist
Robots performing delicate brain surgeries need to make very precise and safe movements. The authors created SurgiCLAW, a system that uses a language model to decide the right small surgical steps by understanding images during surgery. It then relies on reliable control programs to carry out these steps accurately. Tested on actual robots, SurgiCLAW was able to pick the right actions most of the time and complete complex surgical tasks successfully. This approach combines smart decision-making with guaranteed accurate execution.
Open → 2610.15766v1

Surgcast predicts surgical tool movement to control video generation

SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models

Abstract: Surgical world models must turn prospective robot actions into visually faithful, structurally consistent instrument motion. We introduce SurgCast, which predicts future instrument skeletons from current structure, robot state, and future actions, then uses the forecast to control video generation. Dual-path injection combines skeleton geometry with derived, instrument-specific motion descriptors through geometric residuals and conditioning modulation. Both paths complement native action conditioning and are retained in a four-step causal student. On SutureBot and SRTH-Porcine-Cholei, the teacher and student improve all seven reported quality and instrument-consistency metrics over their respective direct-action baselines; teacher FVD decreases by 37.6% and 67.3%. Ablations show that future actions improve skeleton prediction and that combining both injection paths performs best across all six metrics under predicted skeletons. These results support using predicted articulated structure and its motion semantics as a control interface for surgical video prediction. Project videos and examples are available at https://surgcast.netlify.app/.

Sun 11 OctComputer Vision and Pattern Recognition
The gist
In surgery, it’s important to predict how robotic tools will move next to create realistic video simulations. The authors introduce SurgCast, a system that forecasts future positions of surgical tool skeletons based on current state and planned actions. SurgCast then uses these predictions to generate more accurate and consistent surgical videos. Their tests show that predicting future tool motion helps improve the quality and reliability of these video models.
Open → 2610.15645v1

Observation anchored forecasting improves tissue deformation prediction

Observation-Referenced Tissue Deformation Forecasting with Tool-Driven and Residual Dynamics

Abstract: Forecasting tissue deformation from future instrument motion could support predictive planning and increasingly autonomous surgical systems. We study this problem using a scene-specific differentiable surface model fitted to an observed tool-tissue interaction window. Rather than forecasting absolute simulated positions, we introduce an \textit{observation-referenced} formulation that anchors each tissue point to its last valid observation and adds only the deformation induced by the known, preplanned future tool trajectory. This reduces sensitivity to reconstruction, initialization, and persistent simulation offsets while preserving the predicted deformation response. The physical model uses differentiable, mesh-free position-based dynamics with scene-specific rigidity, attachment, region scale, and grasp coupling. We further introduce an optional \textit{observation-referenced residual model} combining graph-spectral spatial modes with periodic harmonic dynamics for structured motion not explained by the tool input. We evaluate the framework retrospectively on 12 surgical manipulation clips from three datasets. Observation referencing provides the largest gain, with a median within-clip error ratio of $0.598$ relative to the corresponding absolute simulation forecast. The optional periodic residual further improves forecasting when the remaining motion exhibits periodic structure. These results support observation-referenced deformation prediction as a practical building block for future surgical world models, digital twins, and generative simulators that must predict how tissue evolves under instrument actions.

Fri 9 OctRobotics
The gist
Predicting how soft tissues change shape when surgical tools move is important for planning surgeries and advanced robotic assistance. This paper introduces a new way to predict tissue movement by referencing each point to its last seen position and adding predicted changes from known tool movements. This approach reduces errors from mismatches in initial setups and ongoing simulation issues. The authors also add a model to capture repetitive tissue motions that tools don't cause. Their tests on surgical videos show this method improves accuracy, making it useful for future surgical planning and simulation tools.
Open → 2610.14191v1

Activereg improves bone registration with fewer probing points

ActiveReg: Information-Driven Active Regional Probing for Partial-to-Full Bone Registration

Abstract: Accurate bone registration is essential for orthopedic surgical navigation and robotic assistance. In a common workflow, the surgeon needs to identify and probe a number of prescribed locations on the exposed bone surface to obtain data points for registration, which can be difficult and time-consuming under limited surgical exposure. Reducing the number of required points while providing clear probing guidance would ease the surgeon's acquisition task intra-operatively. We present ActiveReg, a closed-loop framework that recommends probing regions rather than individual points, allowing flexibility in the exact contact location. A D-optimal planner that takes into account the already acquired points and the uncertainty of current registration information helps achieve accurate registration with fewer points. An online assessment combines pre-update innovation consistency with uncertainty at predefined surgical task locations improves the reliability of completion decision, without requiring ground truth. Simulations across four anatomical scenarios demonstrated that ActiveReg can achieve competitive accuracy compared with baseline registration methods while using substantially fewer acquired points. Real phantom experiments with optical and electromagnetic tracking demonstrated comparable mean target registration error to the Gradient-SDF framework, using only 24-28 acquired points instead of 771-1263.

Wed 7 OctRobotics
The gist
Surgical procedures often require pinpointing spots on bones to help guide navigation and robotic tools, but this can be slow and tricky when there’s limited access. The authors offer ActiveReg, a system that suggests regions instead of single points for probing, making it easier and more flexible for surgeons. It smartly picks where to probe next based on what’s already known, reducing how many points need to be touched while still staying accurate. Tests on simulated models and real setups showed it works well with much fewer points than earlier methods.
Open → 2610.11003v1

Ultra-wideband ranging tracks laparoscopic tool tips behind drapes

Drape-Compatible Tool-Tip Localization for Hand-Held Laparoscopic Instruments via UWB Carrier-Phase Ranging and Trocar-Constrained Geometry

Abstract: A surgical robot policy needs to know where each instrument's working end sits relative to the camera and to the other instrument, yet a laparoscopic operation leaves only the endoscope video, and a draped instrument hides every optical path on itself. We estimate the tool tips from distances and an IMU alone. The distances are measured by ultra-wideband carrier phase between antenna nodes on the handle side of the instruments: one per hand-held instrument, one on the endoscope, all outside the sterile barrier. Take the endoscope antenna as the reference. The two instrument antennas carry six coordinates while the three pairs give three distances, so the problem is short by three, and averaging cannot close that gap because what is missing is information, not precision. Carrier phase adds one more unknown per pair, an integer that leaves distance fixed only modulo lambda/2 = 23.1 mm. The trocar settles both difficulties: each shaft passes through a port whose position is known and whose direction the on-board IMU measures, so its antenna keeps a single degree of freedom, the insertion depth. Two unknowns now face three distances -- two fix the depths and the third is left over, and that spare equation exposes a wrong integer or a wrong mounting offset instead of absorbing it. The same solve returns both tool tips in the endoscope frame, without the second carrier frequency that integer resolution usually needs, so the link never leaves a single PLL lock. In an 11-hole phantom among metal instruments the three distances hold sigma = 0.10-0.33 mm at 35-46 Hz, a sterile drape pressed onto the antennas leaves the link unchanged where an optical path would be blocked, and the logger was carried into a live rabbit appendectomy.

Wed 30 SeptRobotics
The gist
Laparoscopic surgery requires knowing exactly where each tool’s tip is, but cameras can’t see instruments covered by sterile drapes. The authors developed a way to locate these tips using radio signals and motion sensors on the tool handles outside the sterile field. By fixing the tools’ paths through known entry points and combining distance measurements, their method figures out the position of the tips with millimeter accuracy. This lets surgeons and robots track tools even when cameras are blocked.
Open → 2609.39129v1

SurgGMF forecasts future surgical scenes using Gaussian motion fields

SurgGMF: Fully Causal Gaussian Motion Forecasting for Anticipatory Surgical Scene Rendering

Abstract: Dynamic surgical scene modeling is essential for robotic perception, simulation, and decision support. Although existing neural rendering methods enable efficient reconstruction and rendering of deformable surgical scenes, they remain primarily focused on observed-frame reconstruction rather than forecasting future scene states. To this end, we present SurgGMF, a fully causal Gaussian motion forecasting framework for anticipatory surgical scene rendering. Rather than predicting future RGB images directly, SurgGMF forecasts future Gaussian motion states represented by position, scale, and rotation residuals (X/S/R) from historical Gaussian motion fields. To prevent target leakage, we introduce a full-causal-last rendering protocol, where future Gaussian states are rendered without accessing target-frame Gaussian attributes while preserving causal appearance propagation. We evaluate SurgGMF on 12 EndoNeRF and StereoMIS video slices using neural temporal learners and classical dynamics baselines under a unified forecasting protocol. Learned Gaussian motion forecasting consistently outperforms classical dynamics baselines in render space, demonstrating gains beyond hand-crafted state extrapolation. Latency analysis further reveals an accuracy--efficiency trade-off: under the current implementations, TKAN achieves the highest accuracy, whereas GRU and LSTM provide more favorable module-level latency profiles. These results establish SurgGMF as a reproducible framework for causal Gaussian motion forecasting and advance surgical Gaussian representations from retrospective reconstruction toward predictive scene modeling.

Mon 28 SeptComputer Vision and Pattern Recognition
The gist
Predicting what will happen next in a surgical scene can help robots assist better during operations. The authors created SurgGMF, a method that predicts future movement in surgery videos using a special way to represent motion called Gaussian motion fields. Instead of guessing future video frames directly, SurgGMF forecasts position, size, and rotation changes to show how things will move and appear next. Their method works better than traditional motion prediction techniques, allowing for more accurate and efficient anticipation of surgical scenes.
Open → 2609.34733v1

SurgFlow improves surgical robot tool targeting using 3D object motion

SurgFlow: 3D Object-Centric Contact Flow for Surgical Robot Manipulation

Abstract: Paired video-action demonstrations enable autonomous surgical behavior, but such data is scarce: robots perform roughly 1% of surgeries, while video-only data is abundant. Learning 3D object flow offers an embodiment-agnostic way to utilize video data, but flow alone specifies how an object should move, not where and when the tool should engage it, a distinction that is critical in surgery. We introduce SurgFlow, a framework that learns 3D Object-Centric Contact Flow from stereo surgical video without action labels. For each object point, it predicts a future 3D trajectory and contact scores. We extract targets via 3D tracking and tool-object proximity, train a flow matching generator to predict them, and use predicted contact to trigger grasp and release while optimizing end effector motion from flow. On the da Vinci Research Kit (dVRK), SurgFlow succeeds in 37 of 39 stage evaluations across tissue retraction, bimanual reveal, needle pickup, and handover, outperforming baselines trained on equal data with or without action labels. Zero-shot transfer to a humanoid-based laparoscopic robot achieves 85% and 70% average success under similar and novel camera viewpoints, respectively.

Sun 27 SeptRobotics
The gist
Surgical robots need to know not just how objects move but also when and where to touch them during surgery, which is a tough problem. The authors created SurgFlow, a system that learns from stereo surgery videos to predict both 3D object movements and when surgical tools should contact those objects without needing labeled actions. SurgFlow was tested on a common surgical robot, where it succeeded in almost all tasks, and it also worked well when transferred to a different robot and camera views. This approach helps surgical robots better understand and interact with tissues during procedures using widely available video data.
Open → 2609.33237v1

Large diverse surgical video dataset improves instrument segmentation

LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation

Abstract: Referring surgical video instrument segmentation (RSVIS) aims at segmenting the instrument in a surgical video, given a textual description. Despite recent progress, current models are trained and assessed on relatively small-scale benchmarks, hindering the development of more general RSVIS. In addition, existing benchmarks support only the single-target expression that refers to one instrument in the video, while overlooking multi-target and no-target referring expressions, restricting the applicability of RSVIS in practical scenarios. Addressing these issues, we propose LD-RSVIS, a new benchmark aiming to facilitate more robust and general RSVIS. Specifically, LD-RSVIS consists of 3,536 surgical videos with 1.09 million frames and covers a broad set of 30 instrument classes from 25 various procedures. By including abundant videos and classes, LD-RSVIS could benefit large-scale training and evaluation of more general RSVIS methods. Besides, unlike existing datasets, LD-RSVIS offers diverse referring settings, including no-target, single-target, and multi-target expressions, which enables the development of more practical RSVIS models in real applications. In order to ensure high-quality annotations, all videos in LD-RSVIS are manually labeled with multiple rounds of inspection and refinement. To our knowledge, LD-RSVIS is the largest and most diverse benchmark for RSVIS. To analyze LD-RSVIS and to provide comparison for future research, we evaluate 12 representative methods, and the results reveal that more efforts are required for improvements. To encourage future research, we present a simple yet effective RSVIS method, dubbed Cascade-RSVIS, that first mines target-specific cues using the complementary multi-cue text information and then employs such cues and textual information for segmentation, achieving promising performance. Our benchmark and code will be released.

Sat 19 SeptComputer Vision and Pattern Recognition
The gist
Surgical videos often include lots of instruments, and doctors or computers need to identify these tools from descriptions. The authors found that existing datasets are small and limited, only referring to one instrument at a time. They created a much bigger dataset with many videos, instrument types, and different ways to describe the instruments, making it easier to train and test better computer models. They also tested several existing models and introduced a new method that showed promising results.
Open → 2609.23067v1

Self supervised video tracking improves surgery without annotations

S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks

Abstract: Robust point tracking in endoscopic videos is essential for computer-assisted intervention and autonomous robotic surgery, enabling continuous registration between intraoperative video and preoperative imaging despite soft tissue deformation. However, supervised tracking methods depend on large annotated datasets, while surgical conditions make reliable trajectory annotation challenging. We propose a self-supervised Track-Any-Point approach that learns from unlabeled surgical videos by establishing global pixel correspondences and inferring point trajectories through contrastive random walks. Trained without annotations, our method achieves performance comparable to existing semi-supervised approaches while implicitly handling tissue deformation. These findings demonstrate the feasibility of self-supervised point tracking in surgical environments and its potential to reduce reliance on annotated data.

Sun 13 SeptComputer Vision and Pattern RecognitionMachine LearningRobotics
The gist
Tracking points on soft and moving tissue during surgery is very important but hard to teach computers because it's difficult to label videos with exact point movements. The authors created a method that learns to track points in surgical videos without needing labeled examples by figuring out pixel matches across frames on its own. This method performs about as well as approaches that use some supervision, and it naturally handles tissue changes during surgery. Their work shows it's possible to track surgical tissue points accurately without expensive annotations.
Open → 2609.14313v1