Method and system of hand segmentation and overlay using depth data

Patent metadata

Field	Value
Publication number	US-9858475-B2
Application number	US-78055710-A
Country	US
Kind code	B2
Filing date	May 14, 2010
Priority date	May 14, 2010
Publication date	Jan 2, 2018
Grant date	Jan 2, 2018

Abstract

In a minimally invasive surgical system, a plurality of video images is acquired. Each image includes a hand pose image. Depth data for the hand pose image is also acquired or synthesized. The hand pose image is segmented from the image using the depth data. The segmented image is combined with an acquired surgical site image using the depth data. The combined image is displayed to a person at a surgeon's console of the minimally invasive surgical system. Processing each of the video images in the plurality video images in this way reproduces the hand gesture overlaid on the video of the surgical site in the display.

First claim

We claim:

1. A method comprising:
receiving an acquired video image at a controller, wherein the acquired video image comprises a hand pose image of a hand gesture;
segmenting, by the controller, the hand pose image from the acquired video image to obtain a segmented hand pose image, the segmenting comprising a depth threshold process and a flood fill process, the depth threshold process and the flood fill process each using depth data for pixels in the acquired video image, the depth threshold process eliminating pixels having a depth greater than a maximum depth threshold from the acquired video image to obtain a first modified data frame, and the flood fill process processing the first modified data frame to obtain a hand pose mask;
combining, in real time by the controller, the segmented hand pose image with an image of a surgical site to obtain a combined image, wherein the combining comprises using an alpha mask; and
sending the combined image from the controller to a display device,
wherein the display device is included in a console;
wherein the console comprises a master manipulator; and
wherein the segmenting the hand pose image further comprises the controller using information characterizing the master manipulator in the segmenting, wherein the information is different from depth data.

2. The method of claim 1 , wherein the information includes a static image of the master manipulator.

3. The method of claim 1 , wherein the information includes kinematic data for a position of the master manipulator.

4. A method comprising:
receiving an acquired video image at a controller, wherein the acquired video image comprises a hand pose image of a hand gesture;
segmenting, by the controller, the hand pose image from the acquired video image to obtain a segmented hand pose image, the segmenting comprising a depth threshold process and a flood fill process, the depth threshold process and the flood fill process each using depth data for pixels in the acquired video image, the depth threshold process eliminating pixels having a depth greater than a maximum depth threshold from the acquired video image to obtain a first modified data frame, and the flood fill process processing the first modified data frame to obtain a hand pose mask;
combining, in real time by the controller, the segmented hand pose image with an image of a surgical site to obtain a combined image, wherein the combining comprises using an alpha mask; and
sending the combined image from the controller to a display device,
wherein the display device is included in a console;
wherein the console comprises a master manipulator; and
wherein the method further comprises:
parking the master manipulator to provide an unobstructed volume in which to make the hand gesture.

Assignees

• Itkowitz Brandon D
• Dimaio Simon P
• Zhao Tao
• Intuitive Surgical Operations

Inventors

• Itkowitz Brandon D
• Dimaio Simon P
• Zhao Tao

Classifications

• G06T2207/20112

  Image segmentation details · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06K9/00375 Primary

  Physics · mapped topic

• G06V40/107 Primary

  Static hand or arm · CPC title