Augmented reality image display system and surgical robot system comprising the same

Patent metadata

Field	Value
Publication number	US-9767608-B2
Application number	US-201314132782-A
Country	US
Kind code	B2
Filing date	Dec 18, 2013
Priority date	Mar 13, 2013
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

An augmented reality image display system may be implemented together with a surgical robot system. The surgical robot system may include a slave system performing a surgical operation, a master system controlling the surgical operation of the slave system, an imaging system generating a virtual image of the inside of a patient's body, and an augmented reality image display system including a camera capturing a real image having a plurality of markers attached to the patient's body or a human body model. The augmented reality image system may include an augmented reality image generator which detects the plurality of markers in the real image, estimates the position and gaze direction of the camera using the detected markers, and generates an augmented reality image by overlaying a region of the virtual image over the real image, and a display which displays the augmented reality image.

First claim

What is claimed is:

1. A surgical robot system, comprising:
a slave system including a controller configured to perform a surgical operation upon an object, the slave system including an endoscope configured to capture an interior image of a region inside of the object that includes at least a portion of a surgical tool;
a master system including a controller configured to control the surgical operation of the slave system by providing control signals to the slave system; and
an imaging system including a controller configured to generate a virtual image of an inside of the object; and
an augmented reality image display system including a camera, a display and a controller configured to,
control the camera to capture a real image of the object and a plurality of markers attached to the object,
detect a plurality of markers in the real image,
estimate a position and gaze direction of the camera using the detected markers,
generate an augmented reality image by overlaying a region of the virtual image corresponding to the estimated position and gaze direction of the camera over the real image, generating a virtual tool corresponding to the surgical tool, and combining the virtual tool and the interior image of the surgical tool captured by the endoscope with the virtual image such that the surgical tool and the virtual tool overlap within the augmented reality image, and
control the display to display the augmented reality image.

2. The surgical robot system according to claim 1 , wherein the slave system comprises:
the surgical tool, the surgical tool being a tool for performing the surgical operation;
a position sensor to detect a position of the surgical tool; and
a position calculator to calculate position information of the surgical tool using detected signals by the position sensor.

3. The surgical robot system according to claim 2 , wherein the controller of the augmented reality image display system is configured to receive position information of the surgical tool from the slave system and generate the virtual tool at a region matching the position information in the augmented reality image.

4. The surgical robot system according to claim 1 , wherein:
the camera is attached to the display; and
the controller of the augmented reality display system is configured to generate the augmented reality image by estimating the position and gaze direction of the camera changing in accordance with movement of the display in real-time and by compositing a region of the virtual image corresponding to the estimated position and gaze direction of the camera and the real image.

5. The surgical robot system according to claim 1 , wherein the controller of the augmented reality display system is configured to calculate position information of each of the detected markers in the real image and estimate the position and gaze direction of the camera using the calculated position information of each of the markers.

6. The surgical robot system according to claim 5 , wherein the position information of each of the markers includes a distance between the markers and a connection angle between the markers.

7. The surgical robot system according to claim 5 , wherein the position information of each of the markers includes size information of a first marker from among the plurality of markers in the real image.

8. The surgical robot system according to claim 7 , wherein the controller of the augmented reality image display system is configured to generate the augmented reality image by calculating a distance between the camera and the first marker using the size information of the first marker, enlarge or contracts the virtual image in accordance with the calculated distance, and composite the virtual image and the real image.

9. The surgical robot system according to claim 1 , wherein the controller of the imaging system is configured to,
convert an image of the object captured in advance of the surgical operation, into a 3-D image;
generate the virtual image by projecting the converted 3-D image onto the interior image captured by the endoscope; and
store the 3-D image and the virtual image.

10. The surgical robot system according to claim 9 , wherein the image of the object captured in advance of the surgical operation includes at least one of a computed tomography (CT) image and a magnetic resonance imaging (MRI) image.

11. The surgical robot system according to claim 1 , wherein the augmented reality image is generated to have a region corresponding to the position and gaze direction of the camera to face forward.

12. An augmented reality image display system comprising:
a camera configured to capture a real image of an object and a plurality of markers attached to the object;
a controller configured to,
detect a plurality of markers in the real image,
estimate a position and gaze direction of the camera using the detected markers, and
generate an augmented reality image by overlaying a region of a virtual image corresponding to the estimated position and gaze direction of the camera over the real image, generating a virtual tool corresponding to a surgical tool, and combining the virtual tool and an image of the surgical tool with the virtual image such that the surgical tool and the virtual tool overlap within the augmented reality image, the image of the surgical tool being an image captured inside the object by an endoscope; and

a display configured to display the augmented reality image.

13. The augmented reality image display system according to claim 12 , wherein:
the camera is attached to the display; and
the controller of the augmented reality image display system is configured to generate the augmented reality image by estimating the position and gaze direction of the camera changing in accordance with movement of the display in real-time and by compositing a region of the virtual image corresponding to the estimated position and gaze direction of the camera and the real image.

14. The augmented reality image display system according to claim 12 , wherein the controller of the augmented reality image display system is configured to calculate position information of each of the detected markers in the real image and estimate a position and gaze direction of the camera using the calculated position information of each of the markers.

15. The augmented reality image display system according to claim 14 , wherein the position information of each of the markers includes a distance between the markers and a connection angle between the markers.

16. The augmented reality image display system according to claim 14 , wherein the position information of each of the markers includes size information of a first marker from among the markers in the real image.

17. The augmented reality image display system according to claim 16 , wherein the controller of the augmented reality image display system is configured to,
generate the augmented reality image by calculating a distance between the camera and the first marker using the size information of the first marker,
enlarge or contract the virtual image in accordance with the calculated distance, and
composite the virtual image and the real image.

18. The augmented reality image display system according to claim 12 , wherein the augmented reality image includes the virtual image of the surgical tool inserted into the object.

19. The augmented reality image display system according to claim 12 , wherein the augmented reality image is generated to have a region correspondin

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Hee Kuk
• Hwang Won Jun
• Roh Kyung Shik
• Choi Jung Yun

Classifications

• A61B1/04

  combined with photographic or television appliances · CPC title

• A61B34/30

  Surgical robots · CPC title

• A61B2034/2065

  Tracking using image or pattern recognition · CPC title

• A61B2090/3937

  Visible markers · CPC title

• A61B2090/365

  augmented reality, i.e. correlating a live optical image with another image · CPC title