Papers for

medical simulation developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Soft haptic device changes both temperature and stiffness simultaneously

A Multimodal Haptic Interface for Simultaneous Softness and Thermal Modulation

Abstract: Haptic interactions, such as touching an object in virtual reality, involve detecting a plethora of sensations through the skin. Whilst tactile displays exist to simulate and manipulate all of these cues individually, little progress has been made constructing devices that can control multiple cues simultaneously. This paper presents a tactile display technology that offers both softness and temperature change. A prototype display, based on particle jamming and the thermal transfer of heated and cooled water, is evaluated in terms of its ability to change both stiffness and temperature. Mechanical evaluations demonstrate controllable stiffness increase of 300%, whilst thermal characterisation showed strong surface uniformity and a range of achievable temperatures from 10°C up to 50°C, with closed-loop control maintaining a 0.2°C error from a setpoint. This work advances the design of multimodal tactile displays with applications to perceptual research, medical simulation, and affective haptics.

Thu 5 NovRoboticsHuman-Computer Interaction
The gist
Touching things feels more real when you can sense both how soft or hard they are and whether they feel warm or cool. The authors created a device that can change softness and temperature at the same time using special particles and water that can be heated or cooled. Their tests showed it can make the surface much stiffer and control temperatures precisely from cold to warm. This technology could help virtual reality feel more lifelike and improve training tools that rely on touch.
Open → 2611.08398v1

Automated estimation of complete patient bodies from partial scans

MedBOE: Optimization-Based Estimation of Complete Patient Body Models from Whole-Body Medical Imaging

Abstract: True whole-body medical imaging captures a patient from head to toe and provides comprehensive anatomical and/or functional information. This coverage enables reconstruction of the external body surface, yielding patient body models. However, scanner geometry (e.g., bore diameter), patient characteristics, and positioning can leave body regions missing (e.g., lateral arms/torso, hands, forearms, elbows), producing incomplete patient body models. Such incompleteness limits downstream medical analyses, including derivation of body-habitus metrics (BHMs), such as body volume and surface area, for injected radiotracer activity regimens, cardiometabolic risk assessment, and computational anatomical modeling for medical imaging simulation and radiation dosimetry. We present the Medical Body model Optimization-based Estimator (MedBOE), an automated, training-free framework for estimating complete patient body models from whole-body images with missing body regions. Our innovation is to leverage the Skinned Multi-Person Linear (SMPL) template, a parametric body mesh representing a broad range of body shapes and poses, in medical imaging. MedBOE aligns the SMPL template to the incomplete patient body models via attribute-guided optimization and estimates missing body regions through geometrically and volumetrically constrained deformation. We evaluated MedBOE using two public datasets: AutoPET (91 whole-body PET-CT images) and Human Implicit Tissues (HIT; 40 whole-body segmentations from MRI images). MedBOE achieved the best overall volumetric and geometric performance among evaluated methods. BHMs derived from MedBOE were within 2% error, suggesting low percentage errors for downstream analyses. MedBOE offers an automated, training-free method for estimating complete patient body models from whole-body images with missing body regions.

Tue 20 OctComputer Vision and Pattern Recognition
The gist
Whole-body medical scans sometimes miss parts of a patient's body due to limitations like scanner size or patient positioning. The authors present MedBOE, a new method that can fill in these missing body parts automatically without needing prior training. MedBOE uses a flexible 3D body model to match the visible parts and then estimates the missing regions based on shape and volume constraints. Tests on public datasets show it produces accurate complete body models, helping with measurements important in medical care.
Open → 2610.28985v1

Vr nursing training boosts student confidence and skill learning

Iterative Design and Evaluation of a VR Nursing Training System: Insights from Clinicians and Student Learners

Abstract: This paper presents the iterative design and evaluation of a virtual reality (VR) nursing training system developed in collaboration with nursing education specialists to enhance procedural skill acquisition, clinical decision-making, and critical thinking among pre-licensure nursing students. Two clinical scenarios, chest pain and oxygen therapy, were created to simulate acute care settings, with design refinements informed by an expert evaluation study (n=8) and further assessed through an effectiveness study involving nursing students (n=17). Results demonstrate high user acceptance, perceived usefulness, and significant post-simulation increases in learner confidence. The VR system was found to be an accessible supplemental training tool that can foster self-directed learning. Finally, we present design recommendations for developing VR simulations for nursing education, emphasizing collaboration with domain experts, accommodating diverse learning goals, and simulating complex clinical workflows.

Tue 13 OctHuman-Computer Interaction
The gist
Learning nursing skills can be hard because real-life practice is limited. The authors worked with nursing experts to build a virtual reality (VR) tool that simulates hospital scenarios like chest pain and oxygen therapy. Nursing students who used the VR system felt more confident and found it useful for practicing on their own. The authors also gathered expert feedback to improve the system step by step.
Open → 2610.19152v1

AphaVoice generates realistic aphasia speech for medical training

AphaVoice: Text-to-Speech Model for Aphasia Patient Simulation

Abstract: While language therapy is the primary treatment for post-stroke aphasia, limited clinical resources often restrict patient access to proper care. Addressing this gap requires reliable training simulations for medical professionals. However, current AI efforts focus almost exclusively on speech processing, leaving speech generation largely unexplored. This study introduces AphaVoice, a novel text-to-speech (TTS) framework explicitly engineered to synthesize highly realistic aphasic speech. Evaluated against state-of-the-art TTS systems, AphaVoice demonstrates a superior ability to capture the clinical feature of dysfluency, outperforming existing models in temporal metrics while consistently maintaining speaker identity. By leveraging latent space shifting, the model also supports multi-speaker generation without sacrificing phonetic realism. Ultimately, AphaVoice provides a scalable, high-fidelity tool for medical training and clinical simulation related to aphasia.

Sat 10 OctComputation and LanguageArtificial Intelligence
The gist
People recovering from strokes sometimes have trouble speaking clearly, which makes it hard for doctors to practice helping them. The authors created a computer system called AphaVoice that can read text aloud in a way that mimics the speech patterns of people with aphasia. This system sounds realistic by capturing speech disruptions and keeping the speaker's voice consistent. It can even produce different speaker voices while still sounding like someone with aphasia. This could help train medical professionals more effectively.
Open → 2610.14479v1

Observation anchored forecasting improves tissue deformation prediction

Observation-Referenced Tissue Deformation Forecasting with Tool-Driven and Residual Dynamics

Abstract: Forecasting tissue deformation from future instrument motion could support predictive planning and increasingly autonomous surgical systems. We study this problem using a scene-specific differentiable surface model fitted to an observed tool-tissue interaction window. Rather than forecasting absolute simulated positions, we introduce an \textit{observation-referenced} formulation that anchors each tissue point to its last valid observation and adds only the deformation induced by the known, preplanned future tool trajectory. This reduces sensitivity to reconstruction, initialization, and persistent simulation offsets while preserving the predicted deformation response. The physical model uses differentiable, mesh-free position-based dynamics with scene-specific rigidity, attachment, region scale, and grasp coupling. We further introduce an optional \textit{observation-referenced residual model} combining graph-spectral spatial modes with periodic harmonic dynamics for structured motion not explained by the tool input. We evaluate the framework retrospectively on 12 surgical manipulation clips from three datasets. Observation referencing provides the largest gain, with a median within-clip error ratio of $0.598$ relative to the corresponding absolute simulation forecast. The optional periodic residual further improves forecasting when the remaining motion exhibits periodic structure. These results support observation-referenced deformation prediction as a practical building block for future surgical world models, digital twins, and generative simulators that must predict how tissue evolves under instrument actions.

Fri 9 OctRobotics
The gist
Predicting how soft tissues change shape when surgical tools move is important for planning surgeries and advanced robotic assistance. This paper introduces a new way to predict tissue movement by referencing each point to its last seen position and adding predicted changes from known tool movements. This approach reduces errors from mismatches in initial setups and ongoing simulation issues. The authors also add a model to capture repetitive tissue motions that tools don't cause. Their tests on surgical videos show this method improves accuracy, making it useful for future surgical planning and simulation tools.
Open → 2610.14191v1

Differentiable 3d model links visual data to solid mechanics simulation

A differentiable Lagrangian-coupled 3D Gaussian Splatting-SPH model for forward simulation and inverse analysis in solid mechanics

Abstract: Recent advances in generative world models have increased interest in digital models that reproduce both the appearance of real objects and their response to physical interaction. Three-dimensional reconstruction techniques, including 3D Gaussian Splatting, capture detailed surface geometry and appearance from images and videos. However, extending these representations beyond plausible animation to mechanically interpretable models for constitutive behavior, boundary conditions, and inverse parameter identification remains less explored. In this work, a differentiable Lagrangian-coupled 3DGS-smoothed particle hydrodynamics (SPH) model is proposed for forward simulation and inverse analysis of deformable solids. The observed object is first reconstructed from multi-view calibrated visual dataset as a 3DGS rendering model. An envelope-based procedure then generates an independent SPH support for the solid-mechanics model, avoiding the direct use of rendering primitives as mechanical particles. A reference-configuration Lagrangian transfer maps SPH deformation to Gaussian positions and covariances, thereby coupling the physical model and the image observation model while preserving a differentiable computational path. The SPH formulation supports linear elastic, hyperelastic, and Kelvin--Voigt viscoelastic responses, together with fixed, free, and Robin-type boundary conditions. Numerical studies validate the SPH response against finite-element results, assess accuracy and efficiency against a conventional model using Gaussian centers as surface SPH particles, and demonstrate forward simulations on beam, bridge, and liver-shaped examples. Inverse analyses further estimate constitutive and boundary parameters from rendered deformation observations, including noisy cases, demonstrating the feasibility of the proposed model for mechanics-based parameter identification from image data.

Sat 3 OctComputer Vision and Pattern Recognition
The gist
Capturing both how objects look and how they behave when touched or deformed is tricky. The authors present a new method that first reconstructs an object’s detailed 3D shape and appearance from images, then models how that object physically deforms using a physics-based method called smoothed particle hydrodynamics (SPH). Their approach connects the visual model and physical simulation in a way that can be differentiated, meaning it can be used to figure out material properties or boundary conditions from observed deformations. They tested their method on examples like beams and shapes resembling organs, showing it can simulate and also estimate physical parameters even with noisy data.
Open → 2610.04336v1

EndoLive enables real-time video style conversion for surgical training

EndoLive: Real-Time Style Transfer for Endoscopic Endonasal Skull Base Surgical Video

Abstract: Complex surgical procedures around critical anatomy, such as the endoscopic endonasal skull base surgery, requires significant practice and training on the part of the surgeon before they are allowed to perform the operation on a live patient. This training in typically done in cadaveric specimens, due to them containing the same critical structures as a living human. However, cadavers are not a perfect 1-to-1 substitute for a living patient. The dead and preserved tissues of a cadaver are colored completely differently than a living human, and -- without complex and expensive pumping systems -- do not bleed in the same way. As a result, identifying the critical pieces of anatomy that make this procedure so complex can be quite different in a live case than in a surgeon's cadaveric practice. This paper presents EndoLive, a framework for real-time style transfer between cadaveric endoscopic video and living human endoscopic video. Our method combines the ConStructS GAN model for realistic style transfer for surgical applications, with the HyPER-GAN model that can learn complex translations and perform them in real-time. We train EndoLive on unpaired cadaveric and live images taken from an endoscope, and test the trained model with cadaveric video, on a variety of devices. Experimental results demonstrate that EndoLive can perform cadaveric-to-live translation at speeds well above the minimum necessary for real-time, while maintaining semantic consistency of critical anatomical structures. Our source code is available at https://github.com/griffhurt/endolive.

Thu 1 OctComputer Vision and Pattern Recognition
The gist
Training surgeons on real human patients is risky, so they practice on cadavers first. But cadaver tissue looks different from living tissue, making it hard to recognize important anatomy during practice. The authors created EndoLive, a system that changes the look of cadaver videos to resemble live surgery videos in real-time. This helps surgeons better prepare by seeing cadaver anatomy as it would appear during actual surgery.
Open → 2610.01956v1

SurgGMF forecasts future surgical scenes using Gaussian motion fields

SurgGMF: Fully Causal Gaussian Motion Forecasting for Anticipatory Surgical Scene Rendering

Abstract: Dynamic surgical scene modeling is essential for robotic perception, simulation, and decision support. Although existing neural rendering methods enable efficient reconstruction and rendering of deformable surgical scenes, they remain primarily focused on observed-frame reconstruction rather than forecasting future scene states. To this end, we present SurgGMF, a fully causal Gaussian motion forecasting framework for anticipatory surgical scene rendering. Rather than predicting future RGB images directly, SurgGMF forecasts future Gaussian motion states represented by position, scale, and rotation residuals (X/S/R) from historical Gaussian motion fields. To prevent target leakage, we introduce a full-causal-last rendering protocol, where future Gaussian states are rendered without accessing target-frame Gaussian attributes while preserving causal appearance propagation. We evaluate SurgGMF on 12 EndoNeRF and StereoMIS video slices using neural temporal learners and classical dynamics baselines under a unified forecasting protocol. Learned Gaussian motion forecasting consistently outperforms classical dynamics baselines in render space, demonstrating gains beyond hand-crafted state extrapolation. Latency analysis further reveals an accuracy--efficiency trade-off: under the current implementations, TKAN achieves the highest accuracy, whereas GRU and LSTM provide more favorable module-level latency profiles. These results establish SurgGMF as a reproducible framework for causal Gaussian motion forecasting and advance surgical Gaussian representations from retrospective reconstruction toward predictive scene modeling.

Mon 28 SeptComputer Vision and Pattern Recognition
The gist
Predicting what will happen next in a surgical scene can help robots assist better during operations. The authors created SurgGMF, a method that predicts future movement in surgery videos using a special way to represent motion called Gaussian motion fields. Instead of guessing future video frames directly, SurgGMF forecasts position, size, and rotation changes to show how things will move and appear next. Their method works better than traditional motion prediction techniques, allowing for more accurate and efficient anticipation of surgical scenes.
Open → 2609.34733v1

Robotic ultrasound navigation predicts anatomy changes with scene graphs

Scanning While Imagining: A Scene-Graph World Model for Robotic Ultrasound Navigation

Abstract: Ultrasound (US) acquisition depends on the operator's ability to interpret anatomy and anticipate how the view will change with probe motion. Many robotic US navigation methods select actions without explicitly predicting these anatomical changes. We propose SonoGraph-WM, an action- and goal-conditioned world model for anticipatory probe navigation. The model represents anatomy as scene graphs (SGs), capturing visible structures, their geometry, and spatial relationships without synthesizing US images. Given a history of SGs and probe poses, a unified Transformer jointly predicts future SGs and poses. A receding-horizon planner recursively imagines candidate trajectories, selects the shortest predicted path reaching a goal graph, and follows it over a short execution horizon before replanning from new observations. To reduce reliance on tracked and anatomically annotated US sequences, we generate aligned SG--pose training data from computed tomography (CT) label maps along surface-constrained probe trajectories. On four held-out CT cases, spatial relation F1 remains above 93% over 20 prediction steps, and closed-loop navigation achieves 77.50% and 75.00% success for the gallbladder and pancreas, respectively, using annotation-derived SGs. In robot--phantom navigation experiments with label-map-derived SGs, the planner reached the target view in 73.7% of trials. These findings support CT-supervised anatomical world modeling for probe planning and highlight the importance of frequent observation updates for reliable navigation. Project Page: https://noseefood.github.io/us-sonograph-wm/

Sat 26 SeptRoboticsArtificial Intelligence
The gist
Ultrasound imaging depends heavily on how well the operator can predict the anatomy seen as the probe moves. The authors created SonoGraph-WM, a system that uses a special map of body structures called scene graphs to predict future views and probe positions without generating images. It uses past data and a planning method to imagine and follow the best path to a target internal structure. They trained and tested their system using CT scans, showing it can reliably navigate robotically to organs like the gallbladder and pancreas. This approach could help make robotic ultrasound scanning more precise and less dependent on manual skill.
Open → 2609.32837v1

Video generation models struggle to simulate medical procedures accurately

REMEDY: How Far Is Video Generation from Medical Education World Models?

Abstract: Recent video generation models produce realistic videos and show potential as a foundation for world models. These advances create opportunities for generating medical teaching demonstrations, which requires both convincing visual quality and precise procedural actions. However, whether current generators can meet these requirements has not been measured. To address this problem, we introduce Readiness Evaluation of Medical Education Demonstration sYnthesis (REMEDY), to our knowledge, the first benchmark for AI-generated medical teaching demonstrations. REMEDY provides 900 first frames from real demonstration videos, covering 12 tasks across four scenarios: operating room, imaging, clinic and bedside, and resuscitation. Five contemporary open-source video generation models produce 4,500 videos from these frames. We combine task-specific clinical checklists with video and motion quality metrics. Evaluation covers four dimensions: clinical action following, clinical profiles, video quality, and motion quality. Our results show that realistic appearance and temporal consistency do not ensure correct clinical actions. Even the most advanced MiniMax-H3 achieves only 28.25% on strict clinical success rate, and fine-grained clinical actions remain challenging. These findings establish a foundation and roadmap for developing future medical education world models.

Sat 26 SeptComputer Vision and Pattern Recognition
The gist
Generating videos that show medical procedures convincingly is very hard because the videos must look real and also show the correct steps. The authors created a new test called REMEDY to check how well AI models can do this. They tested five different video generators on 12 medical tasks but found even the best model got less than 30% of the clinical steps right. This shows current technology is not yet good enough for producing reliable medical teaching videos.
Open → 2609.32460v1

Probabilistic mesh estimation improves deformable object tracking

Differentiable Mesh State Estimation via Factor Graph Inference for Deformable Object Reconstruction

Abstract: Estimating deformable object states remains a fundamental challenge in robotics and simulation. We propose a novel factor graph-based framework for probabilistic mesh state estimation of deformable objects. The method directly updates a tetrahedral mesh, a rich and physically-grounded representation of an environment, by combining physics priors, noisy sensor measurements, and temporal smoothness constraints within a unified probabilistic formulation. The estimation problem is posed as a nonlinear least-squares optimization and solved using Levenberg-Marquardt. Ex vivo central-airway obstruction experiments and simulations on deforming cube models demonstrate reliable and accurate reconstruction under both rigid motion and deformation, highlighting the potential of this probabilistic approach for principled, measurement-driven mesh state estimation in deformable object reconstruction.

Tue 15 SeptRoboticsComputer Vision and Pattern Recognition
The gist
Tracking and understanding objects that change shape, like soft or bendable items, is hard for robots and simulations. The authors created a new method that uses a detailed 3D mesh made of tiny tetrahedrons to represent these objects. Their approach combines knowledge about physics, noisy sensor data, and smooth movement over time into a single calculation to figure out the object’s shape accurately. Tests on airway experiments and cube models show their method works well for objects that both move rigidly and deform. This could help robots better see and use soft or flexible things.
Open → 2609.16686v1