Use of augmented reality to assist navigation during medical procedures

Patent metadata

Field	Value
Publication number	US-11058497-B2
Application number	US-201816229629-A
Country	US
Kind code	B2
Filing date	Dec 21, 2018
Priority date	Dec 26, 2017
Publication date	Jul 13, 2021
Grant date	Jul 13, 2021

Abstract

Physicians performing invasive procedures require accuracy and precision when working with surgical tools. Surgical procedures are increasingly becoming minimally invasive, with physicians operating using cameras to view the surgery site and directing their tools through oculars or video displays. Ideally, the physician should be able to perform the invasive procedure while simultaneously observing both the real-time image of the patient and additional data critical for his medical decisions about the manipulation of the surgical tool and the next surgical step. The augmented reality navigation system of the present disclosure provides tool location visibility for invasive procedures through the use of location sensors included on a camera and/or on the tools used during a procedure. A location tracking system determines and monitors the locations of the tools and camera based on the characteristics of signals detected by the sensors and displays informational overlays on images obtained with a camera.

First claim

What is claimed is:

1. A method for providing an augmented reality display of a subject, the method comprising:
obtaining an image from a camera;
identifying a location of the camera based on signals, wherein signals are received from sensors of a location pad in proximity of the subject's head wherein the location pad is not attached to the patient's head and receiving signals with sensors of the location pad, the signals generated by one or more field generators coupled to the camera;
identifying a field of view of the camera based on the location of the camera;
determining pixel value information of pixels corresponding to anatomical features in the field of view of the image;
identifying one of the anatomical features of the subject based on the pixel value information and the location of the camera;
generating one or more overlay features corresponding to the one anatomical feature in the field of view, the overlay features selected from a group comprising an anatomical feature outline of a sinus ostium and a text configured to label the sinus ostium;
identifying the sinus ostium based on detection of a darkened portion of the image;
in response to the identified sinus ostium, generating the outline of the identified ostium and generating the text on a selected location of the image, said selected location based on a variance of an average color value below a predetermined threshold for a plurality of pixels in an area of the image; and
compositing the generated overlay features for the identified sinus ostium with the image from the camera to form a composited image;
displaying the composited image on a display; and
navigating one or more tools within the sinus ostium based upon the composited image.

2. The method of claim 1 , wherein
determining the pixel value information comprises determining at least one of brightness values of the pixels and color values of the pixels corresponding to the anatomical features in the field of view, and
identifying the one anatomical feature of the subject comprises identifying the one anatomical feature based on at least one of: brightness values of pixels of the one anatomical feature relative to pixels of other anatomical feature in the field of view; and color values of pixels of the one anatomical feature in the field of view.

3. The method of claim 1 , wherein detecting the location of a camera comprises:
performing a pre-procedure registration to correlate a relative position and orientation of the location pad in comparison to a position and orientation of the subject.

4. The method of claim 1 , wherein the one or more overlay features are associated with the one or more tools and generating the one or more overlay features comprises:
generating a graphical representation of a tool.

5. The method of claim 4 , wherein generating the graphical representation of the tool comprises:
determining a position of the tool within the field of view of the camera; and
rendering the graphical representation of the tool based on the position of the tool within the field of view of the camera.

6. The method of claim 1 , wherein generating the overlay feature indicative of the anatomical feature comprises:
determining a position of a marker within the field of view of the camera based on the relative position and orientation of the location pad and the position and orientation of the subject, and based on a relative location of the marker and a model of the subject; and
rendering a graphical representation corresponding to the marker based on the position of the marker within the field of view of the camera.

7. The method of claim 6 , wherein the graphical representation corresponding to the marker includes the text and an arrow as indicated by the marker.

8. The method of claim 6 , wherein the graphical representation corresponding to the marker is adjusted in location, form, and/or format, based on image analysis of the image.

9. A system for providing an augmented reality display of a subject, the system comprising:
a workstation; and
a camera,
wherein the workstation is configured to:
obtain an image from the camera;
identify a location of the camera based on signals, wherein signals are received from sensors of a location pad in proximity of the subject's head wherein the location pad is not attached to the patient's head and receiving signals with sensors of the location pad, the signals generated by one or more field generators coupled to the camera;
identify a field of view of the camera based on the location of the camera;
determine pixel value information of pixels corresponding to anatomical features in the field of view of the image;
identify one of the anatomical features of the subject based on the pixel value information and the location of the camera;
generate one or more overlay features corresponding to the one anatomical feature in the field of view, the overlay features selected from a group comprising an anatomical feature outline of a sinus ostium and a text configured to label the sinus ostium;
identify the sinus ostium based on detection of a darkened portion of the image;
in response to the identified sinus ostium, generating the outline of the identified ostium and generating the text on a selected location of the image, said selected location based on a variance of an average color value below a predetermined threshold for a plurality of pixels in an area of the image; and
composite the generated overlay features for the identified sinus ostium with the image from the camera to form a composited image;
display the composited image on a display; and
navigate one or more tools within the sinus ostium based upon the composited image.

10. The system of claim 9 , wherein
determining the pixel value information comprises determining at least one of: brightness values of the pixels corresponding to the anatomical features in the field of view; and color values of the pixels corresponding to the anatomical features in the field of view, and
identifying the one anatomical feature of the subject comprises identifying the one anatomical feature based on at least one of: brightness values of pixels of the one anatomical feature relative to pixels of other anatomical feature in the field of view; and color values of pixels of the one anatomical feature in the field of view.

11. The system of claim 9 , further comprising the location pad, wherein the workstation is configured to detect the location of the camera by:
performing a pre-procedure registration to correlate a relative position and orientation of the location pad in comparison to a position and orientation of the subject.

12. The system of claim 9 , wherein the one or more overlay features are associated with the one or more tools and the workstation is configured to generate the one or more overlay features by:
generating a graphical representation of a tool.

13. The system of claim 12 , wherein the workstation is configured to generate the graphical representation of the tool by:
determining a position of the tool within the field of view of the camera; and
rendering the graphical representation of the tool based on the position of the tool within the field of view of the camera.

14. The system of claim 9 , wherein the workstation is configured to generate the overlay feature indicative of the anatomical feature by:
determining a position of a marker within the field of view of the camera based on the relative position and orientation of the location pad and the position and orientation of the subject, and based on a relative location of the marker and a model of the subj

Assignees

• Biosense Webster Israel Ltd
• Acclarent Inc

Inventors

• Altmann Andres Claudio
• Govari Assaf
• Gliner Vadim
• Fang Itzhak
• Rachli Noam
• Pinsky Yoav
• Bustan Itamar
• Palushi Jetmir
• Dekel Zvi

Classifications

• A61B1/000094

  extracting biological structures · CPC title

• A61B2034/2057

  Details of tracking cameras · CPC title

• A61B2090/365

  augmented reality, i.e. correlating a live optical image with another image · CPC title

• A61B34/20 Primary

  Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis · CPC title

• A61B90/361

  Image-producing devices, e.g. surgical cameras · CPC title