Systems and methods for displaying medical video images and/or medical 3D models

Patent metadata

Field	Value
Publication number	US-11269173-B2
Application number	US-202016892598-A
Country	US
Kind code	B2
Filing date	Jun 4, 2020
Priority date	Aug 19, 2019
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

Computer-assisted surgical systems and methods provide intraoperative playback of and interaction with recorded video images and/or a 3D model while displaying current video images. The methods and related surgical systems involve capturing, by a two-dimensional (2D) or three-dimensional (3D) video camera, current video images and recording the captured video images. A user interface displays the current video images and the recorded video images and/or the 3D model and enables a user to interact with either or both of them.

First claim

What is claimed is:

1. A method for intraoperatively displaying three-dimensional (3D) video images of anatomical structures, the method comprising:
capturing, by a 3D video camera, 3D video images, yielding captured 3D video images;
recording at least a portion of the captured 3D video images, yielding recorded 3D video images;
displaying at least a portion of the recorded 3D video images and current 3D video images captured by the 3D video camera;
generating a 3D model based on the captured 3D video images;
receiving a signal to change a field of view of a recorded 3D video image of the recorded 3D video images;
changing a field of view of the recorded 3D video image based on the 3D model and the received signal;
determining that a difference between the field of view of the recorded 3D video image and a field of view of a current 3D video image is greater than a threshold; and
in response to determining that the difference is greater than the threshold, indicating a direction to move the field of view of the recorded 3D video image so that the field of view of the recorded 3D video image aligns with the field of view of the current 3D video image and displaying a message not to move a surgical tool.

2. The method of claim 1 , wherein changing the field of view includes rotating the field of view, zooming in or out of the field of view, or translating the field of view.

3. The method of claim 1 , further comprising making the 3D model opaque in a current field of view to show internal anatomic structures or a location of previously placed fiducial markers.

4. The method of claim 1 , further comprising:
detecting an instrument in the 3D model;
removing the instrument from the 3D model to obtain a modified 3D model; and
displaying recorded 3D video images based on the modified 3D model.

5. The method of claim 1 , further comprising:
detecting an instrument in the 3D model;
iteratively determining positions of a body and a tip of the instrument;
iteratively recording the determined positions of the body and the tip in association with the 3D model; and
displaying an indication of the determined positions of the body and the tip.

6. The method of claim 1 , wherein the 3D model is a second 3D model,
the method further comprising generating a first 3D model based on computed tomography (CT) or magnetic resonance imaging (MRI) images of a patient,
wherein the second 3D model is generated based on the recorded 3D video images and the first 3D model.

7. The method of claim 6 , further comprising highlighting those portions of the second 3D model that are based on the first 3D model.

8. A method for intraoperatively displaying three-dimensional (3D) video images of anatomical structures, the method comprising:
capturing, by a 3D video camera, 3D video images, yielding captured 3D video images;
recording at least a portion of the captured 3D video images, yielding recorded 3D video images;
displaying at least a portion of the recorded 3D video images and current 3D video images captured by the 3D video camera;
generating a 3D model based on the captured 3D video images;
receiving a signal to change a field of view of a recorded 3D video image of the recorded 3D video images;
changing a field of view of the recorded 3D video image based on the 3D model and the received signal;
determining that a difference between a field of view of the recorded 3D video image and a field of view of a current 3D video image of the current 3D video images is greater than a threshold;
in response to determining that the difference is greater than the threshold, tracking a position of a surgical tool shown in the field of view of the current 3D video images and determining that the surgical tool is approaching an anatomic structure or has moved outside of the field of view of the current 3D video image; and
in response to determining that the surgical tool is approaching an anatomic structure or has moved outside of the field of view of the current 3D video image, generating a notification.

9. The method of claim 8 , wherein changing the field of view includes rotating the field of view, zooming in or out of the field of view, or translating the field of view.

10. The method of claim 8 , further comprising making the 3D model opaque in a current field of view to show internal anatomic structures or a location of previously placed fiducial markers.

11. The method of claim 8 , further comprising:
detecting an instrument in the 3D model;
removing the instrument from the 3D model to obtain a modified 3D model; and
displaying recorded 3D video images based on the modified 3D model.

12. The method of claim 8 , further comprising:
detecting an instrument in the 3D model;
iteratively determining positions of a body and a tip of the instrument;
iteratively recording the determined positions of the body and the tip in association with the 3D model; and
displaying an indication of the determined positions of the body and the tip.

13. A method for intraoperatively displaying three-dimensional (3D) video images of anatomical structures, the method comprising:
capturing, by a 3D video camera, 3D video images, yielding captured 3D video images;
recording at least a portion of the captured 3D video images, yielding recorded 3D video images;
displaying at least a portion of the recorded 3D video images and current 3D video images captured by the 3D video camera;
generating a 3D model based on the captured 3D video images;
receiving a signal to change a field of view of a recorded 3D video image of the recorded 3D video images;
changing a field of view of the recorded 3D video image based on the 3D model and the received signal;
determining that a structure of interest in the recorded 3D video image is at least partially occluded by at least one other anatomic structure in a current field of view;
receiving a signal to view the structure of interest; and
removing or making transparent or semitransparent the at least one other anatomic structure in the current field of view to make the structure of interest visible in response to determining that a structure of interest in the recorded 3D video image is at least partially occluded by at least one other anatomic structure in the current field of view and in response to receiving the signal to view the structure of interest.

14. The method of claim 13 , wherein changing the field of view includes rotating the field of view, zooming in or out of the field of view, or translating the field of view.

15. The method of claim 13 , further comprising:
detecting an instrument in the 3D model;
removing the instrument from the 3D model to obtain a modified 3D model; and
displaying recorded 3D video images based on the modified 3D model.

16. The method of claim 13 , further comprising:
detecting an instrument in the 3D model;
iteratively determining positions of a body and a tip of the instrument;
iteratively recording the determined positions of the body and the tip in association with the 3D model; and
displaying an indication of the determined positions of the body and the tip.

17. The method of claim 13 , wherein the 3D model is a second 3D model,
the method further comprising generating a first 3D model based on computed tomography (CT) or magnetic resonance imaging (MRI) images of a patient,
wherein the second 3D model is generated based on the recorded 3D video images and the first 3D model.

18. The method of claim 17 , further comprising highlighting those portions of the second

Assignees

• Covidien Lp

Inventors

• Komp John W
• Cantrall Irena
• Rossetto Francesca
• Pierce Robert W
• Morrison Fiona A
• Florio Mark N

Classifications

• G02B23/2415 Primary

  Stereoscopic endoscopes · CPC title

• A61B90/361 Primary

  Image-producing devices, e.g. surgical cameras · CPC title

• A61B1/00194

  adapted for three-dimensional imaging · CPC title

• A61B1/000094

  extracting biological structures · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title