Interactive extended-reality apparatuses for robotic surgery

Patent metadata

Field	Value
Publication number	US-12201369-B2
Application number	US-202318182023-A
Country	US
Kind code	B2
Filing date	Mar 10, 2023
Priority date	Mar 10, 2023
Publication date	Jan 21, 2025
Grant date	Jan 21, 2025

Abstract

Methods, apparatuses, and systems for performing robotic surgery in an extended-reality (XR) surgical simulation environment are disclosed. The disclosed systems provide a surgical metaverse in which a surgical robot network receives medical images of a patient. A digital twin is generated from the medical images that enables generation of a virtual surgical simulation environment for performing a surgical procedure. Necessary workflow objects for the procedure are selected and surgical actions are performed on the digital twin. Data of the workflow objects and actions in relation to the digital twin are stored. The surgical robot network generates a surgical workflow based on the workflow objects and actions and compares the surgical workflow to historical workflows to allow adjustment of the workflow in the surgical simulation virtual environment. The workflow, workflow objects, and actions in relation to the digital twin are sent to the surgical robot.

First claim

We claim:

1. A computer-implemented method for performing a surgical procedure by a surgical robot, comprising:
obtaining one or more images for a patient using one or more sensors;
selecting one or more sensory modalities based on the surgical procedure; generating an immersive experiential multisensory extended-reality (XR) environment by associating one or more virtual models of one or more surgical tools and the surgical robot with the one or more images,
wherein the immersive experiential multisensory XR environment comprises a three-dimensional (3D) digital twin of an anatomy of the patient for performing a simulation of the surgical procedure and provides sensory feedback to a user according to the one or more sensory modalities;

displaying, via an electronic display, the 3D digital twin within the immersive experiential multisensory XR environment for viewing by the user;
identifying virtual surgical actions on the 3D digital twin within the immersive experiential multisensory XR environment;
generating a surgical workflow for the surgical robot, the surgical workflow comprising workflow objects for the surgical procedure based on the virtual surgical actions;
adjusting the surgical workflow based on a comparison of the surgical workflow to stored historical workflows so as to avoid one or more adverse surgical events,
wherein the comparison is used to identify the one or more adverse surgical events associated with the workflow objects based on the virtual surgical actions within the immersive experiential multisensory XR environment;

transmitting the adjusted surgical workflow to the surgical robot to configure the surgical robot with the adjusted surgical workflow,
wherein the adjusted surgical workflow comprises the workflow objects and information describing the surgical actions; and

robotically performing, using the surgical robot, one or more surgical actions on the patient according to the adjusted surgical workflow.

2. The method of claim 1 , comprising:
extracting features from the virtual surgical actions on the 3D digital twin, wherein generating the surgical workflow comprises:
generating the workflow objects using a machine learning model based on the features, the machine learning model trained to provide surgical workflows based on the stored historical workflows.

3. The method of claim 2 , wherein the machine learning model is trained using data obtained from the one or more sensors.

4. The method of claim 1 , comprising:
generating controls on a graphical user interface for the virtual surgical actions on the 3D digital twin using the immersive experiential multisensory XR environment, wherein the controls are customized for the anatomy of the patient and the surgical procedure.

5. The method of claim 1 , comprising:
extracting a success rate of the surgical procedure from the stored historical workflows, wherein the surgical workflow is adjusted based on the success rate.

6. The method of claim 1 , comprising:
generating the 3D digital twin based on the one or more images and historical images of the patient.

7. The method of claim 1 , wherein the one or more images are of a combination of types of medical imaging comprising at least two of magnetic resonance imaging (MRI), computed tomography (CT), X-ray, positron emission tomography (PET), ultrasound, arthrography, angiography, fluoroscopy, or myelography.

8. A robotic surgical system for performing a surgical procedure by a surgical robot, comprising:
a non-transitory computer-readable storage medium storing computer instructions, which when executed by one or more computer processors cause the robotic surgical system to:
obtain one or more images for a patient using one or more sensors;
select one or more sensory modalities based on the surgical procedure;
generate an immersive experiential extended-reality multisensory (XR) environment by associating one or more virtual models of one or more surgical tools and the surgical robot with the one or more images,
wherein the immersive experiential multisensory XR environment comprises a 3D digital twin of an anatomy of the patient for performing a simulation of the surgical procedure and provides sensory feedback to a user according to the one or more sensory modalities;

display, via an electronic display, the 3D digital twin within the immersive experiential multisensory XR environment for viewing by the user;
identify virtual surgical actions on the 3D digital twin within the immersive experiential multisensory XR environment;
generate a surgical workflow for the surgical robot, the surgical workflow comprising workflow objects for the surgical procedure based on the virtual surgical actions;
adjust the surgical workflow based on a comparison of the surgical workflow to stored historical workflows so as to avoid one or more adverse surgical events,
wherein the comparison is used to identify the one or more adverse surgical events associated with the workflow objects based on the virtual surgical actions within the immersive experiential multisensory XR environment;

transmit the adjusted surgical workflow to the surgical robot to configure the surgical robot with the adjusted workflow,
wherein the adjusted surgical workflow comprises the workflow objects and information describing the surgical actions; and

robotically perform, using the surgical robot one or more surgical actions on the patient according to the adjusted workflow.

9. The robotic surgical system of claim 8 , wherein the computer instructions cause the robotic surgical system to:
extract features from the virtual surgical actions on the 3D digital twin, wherein the computer instructions to generate the surgical workflow cause the robotic surgical system to:
generate the workflow objects using a machine learning model based on the features, the machine learning model trained to provide surgical workflows based on the stored historical workflows.

10. The robotic surgical system of claim 9 , wherein the machine learning model is trained using data obtained from the one or more sensors.

11. The robotic surgical system of claim 8 , wherein the computer instructions cause the robotic surgical system to:
generate controls on a graphical user interface for the virtual surgical actions on the 3D digital twin using the immersive experiential multisensory XR environment, wherein the controls are customized for the anatomy of the patient and the surgical procedure.

12. The robotic surgical system of claim 8 , wherein the computer instructions cause the robotic surgical system to:
extract a success rate of the surgical procedure from the stored historical workflows, wherein the surgical workflow is adjusted based on the success rate.

13. The robotic surgical system of claim 8 , wherein the computer instructions cause the robotic surgical system to:
generate the 3D digital twin based on the one or more images and historical images of the patient.

14. The robotic surgical system of claim 8 , wherein the one or more images are of a combination of types of medical imaging comprising at least two of magnetic resonance imaging (MRI), computed tomography (CT), X-ray, positron emission tomography (PET), ultrasound, arthrography, angiography, fluoroscopy, or myelography.

15. The method of claim 1 , comprising:
performing the comparison a plurality of times during the surgical procedure to adjust the surgical workflow and prevent the one or more adverse surgical events predicted based on the surgical workflow.

16. A surgical robot for performing a surgical procedure by a

Assignees

• Ix Innovation Llc

Inventors

• Roh Jeffrey
• Esterberg Justin
• Cronin John
• Cronin Seth
• Baker Michael John

Classifications

• A61B2034/104

  Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring · CPC title

• A61B2034/252

  indicating steps of a surgical procedure · CPC title

• A61B2034/105

  Modelling of the patient, e.g. for ligaments or bones · CPC title

• A61B2034/254

  being adapted depending on the stage of the surgical procedure · CPC title

• A61B2090/367

  creating a 3D dataset from 2D images using position information · CPC title