Method, apparatus, and system for utilizing augmented reality to improve surgery

Patent metadata

Field	Value
Publication number	US-2018140362-A1
Application number	US-201615564347-A
Country	US
Kind code	A1
Filing date	Apr 5, 2016
Priority date	Apr 7, 2015
Publication date	May 24, 2018
Grant date	—

Abstract

A method, apparatus, and computer readable medium are provided for utilizing augmented reality visualization to assist surgery. An example method includes generating a three dimensional reconstruction of an image stack representing a target area of a patient, and superimposing, by a head-mounted display, a projection of the three dimensional reconstruction onto a field of view of a user. The method further includes maintaining alignment between the projection and the user's actual view of the target area using a plurality of fiducial markers associated with the target area. In some embodiments, the method further includes scanning the target area to generate the image stack.

First claim

1 . A method for utilizing augmented reality visualization in surgery, the method comprising:
generating a three dimensional reconstruction of an image stack representing a target area of a patient by
performing image segmentation on all images of the image stack to identify structures within the image stack, and
generating a three dimensional mesh defining boundaries of the identified structures, wherein the three dimensional reconstruction comprises the three dimensional mesh;
superimposing, by a head-mounted display, a three dimensional projection of the three dimensional reconstruction onto a field of view of a user; and maintaining alignment between the projection and the user's actual view of the target area using a plurality of fiducial markers associated with the target area.

2 . The method of claim 1 , further comprising:
scanning the body part to generate the image stack.

3 . The method of claim 2 , where scanning the body part comprises:
performing a computed tomography (CT) scan of the target area; performing an magnetic resonance imaging (MRI) scan of the target area; performing a positronic emission tomography (PET) scan of the target area; or performing a single-photon emission computed tomography (SPECT) scan of the target area.

4 . The method of claim 1 , wherein generating the three dimensional reconstruction of the image stack representing the target area includes:
pre-processing the image stack.

5 . The method of claim 4 , wherein pre-processing the image stack includes at least one of:
aligning images of the image stack; or filtering images of the image stack.

6 . (canceled)

7 . The method of claim 1 , further comprising:
in an instance in which a voxel size of the three dimensional mesh is not isotropic, rescaling the three dimensional mesh.

8 . The method of claim 1 , wherein superimposing the three dimensional projection of the three dimensional reconstruction onto the field of view of the user includes:
displaying, by the head-mounted display, a first projection of the three dimensional reconstruction to one eye of the user; and displaying, by the head-mounted display, a second projection of the three dimensional reconstruction to the other eye of the user, wherein differences between the first projection and the second projection are designed to generate the three dimensional projection of the three dimensional reconstruction.

9 . The method of claim 1 , wherein maintaining alignment between the projection and the user's actual view of the target area includes:
identifying, by a camera associated with the head-mounted display, a location of each of the plurality of fiducial markers; calculating a relative location of the target area from the perspective of the head-mounted display; and generating the projection of the three dimensional reconstruction based on the calculated relative location of the target area.

10 . The method of claim 9 , wherein maintaining alignment between the projection and the user's actual view of the target area further includes:
presenting an interface requesting user adjustment of the alignment between the projection and the user's actual view of the target area.

11 . The method of claim 10 , further comprising:
receiving one or more responsive adjustments; and updating the projection of the three dimensional reconstruction in response to receiving the one or more responsive adjustments.

12 . The method of claim 9 , further comprising, in an instance in which at least two of the plurality of fiducial markers is disposed on the target area:
detecting a change in location or orientation of the at least two fiducial markers disposed on the target area, wherein movement of the at least two fiducial markers disposed on the target area indicates a change in location or shape of the target area; computing a deformation of the three dimensional reconstruction based on the detected change; applying the deformation to the three dimensional reconstruction; and updating the projection of the three dimensional reconstruction in response to applying the deformation to the three dimensional reconstruction.

13 . The method of claim 1 , further comprising:
receiving relative position and orientation information regarding a surgical tool; and superimposing, by the head-mounted display, a projection of the surgical tool onto the field of view of the user.

14 . A system for utilizing augmented reality visualization in surgery, the system comprising:
an apparatus configured to generate a three dimensional reconstruction of an image stack representing a target area of a patient by
performing image segmentation on all images of the image stack to identify structures within the image stack, and
generating a three dimensional mesh defining boundaries of the identified structures, wherein the three dimensional reconstruction comprises the three dimensional mesh; and
a head-mounted display configured to superimpose a three dimensional projection of the three dimensional reconstruction onto a field of view of a user, wherein the apparatus is configured to maintain alignment between the projection and the user's actual view of the target area using a plurality of fiducial markers associated with the target area.

15 . (canceled)

16 . The system of claim 15 , where the device configured to scan the target area comprises:
a computed tomography (CT) scanner; a magnetic resonance imaging (MRI) scanner; a positronic emission tomography (PET) scanner; or a single-photon emission computed tomography (SPECT) scanner.

17 - 19 . (canceled)

20 . The system of claim 14 , wherein the apparatus configured to generate the three dimensional reconstruction of the image stack representing the target area is further configured to:
in an instance in which a voxel size of the three dimensional mesh is not isotropic, rescale the three dimensional mesh.

21 . The system of claim 14 , wherein the head-mounted display is configured to superimpose the projection of the three dimensional reconstruction onto the field of view of the user by:
displaying a first projection of the three dimensional reconstruction to one eye of the user; and displaying a second projection of the three dimensional reconstruction to the other eye of the user, wherein differences between the first projection and the second projection are designed to generate the three dimensional projection of the three dimensional reconstruction.

22 . The system of claim 14 , further comprising:
a camera associated with the head-mounted display and configured to identify a location of each of the plurality of fiducial markers, wherein the apparatus is configured to maintain alignment between the projection and the user's actual view of the target area by
calculating a relative location of the target area from the perspective of the head-mounted display, and
generating the projection of the three dimensional reconstruction based on the calculated relative location of the target area.

23 - 24 . (canceled)

25 . The system of claim 22 ,
wherein the head-mounted display is further configured to, in an instance in which at least two of the plurality of fiducial markers is disposed on the target area:
detect a change in location or orientation of the at least two fiducial markers disposed on the target area, wherein movement of the at least two fiducial markers disposed on the target area indicates a change in location

Assignees

• Univ King Abdullah Sci & Tech

Inventors

• Calì Corrado
• Besuchet Jonathan

Classifications

• A61B2017/00694

  with means correcting for movement of or for synchronisation with the body · CPC title

• A61B90/37

  Surgical systems with images on a monitor during operation · CPC title

• A61B34/20

  Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis · CPC title

• A61B34/10

  Computer-aided planning, simulation or modelling of surgical operations · CPC title

• G06T7/73

  using feature-based methods · CPC title