Efficient 3-D telestration for local and remote robotic proctoring

Patent metadata

Field	Value
Publication number	US-9402690-B2
Application number	US-201414450251-A
Country	US
Kind code	B2
Filing date	Aug 3, 2014
Priority date	Dec 31, 2008
Publication date	Aug 2, 2016
Grant date	Aug 2, 2016

Abstract

An apparatus is configured to show telestration in 3-D to a surgeon in real time. A proctor is shown one side of a stereo image pair, such that the proctor can draw a telestration line on the one side with an input device. Points of interest are identified for matching to the other side of the stereo image pair. In response to the identified points of interest, regions and features are identified and used to match the points of interest to the other side. Regions can be used to match the points of interest. Features of the first image can be matched to the second image and used to match the points of interest to the second image, for example when the confidence scores for the regions are below a threshold value. Constraints can be used to evaluate the matched points of interest, for example by excluding bad points.

First claim

What is claimed is:

1. A medical system comprising:
an image capture device for capturing stereo images of a work site, each stereo image including a first two-dimensional image and a second two-dimensional image, wherein corresponding points in the first and second two-dimensional images depict a common point in the work site and exhibit a disparity between the corresponding points so as to indicate a depth of the common point relative to the image capture device;
a stereo display displaying the first and second two-dimensional images so as to provide a three-dimensional viewing experience; and
a processor that is programmed to: receive information of a first telestration indicative of user selected points of interest of the first two-dimensional image; selectively match the selected points of interest of the first two-dimensional image to corresponding points of the second two-dimensional image to determine matched points of interest of the second two-dimensional image by computing a plurality of image offsets wherein each of the plurality of image offsets is computed using a different image matching method than all others of the plurality of image offsets, and by using one of the computed plurality of image offsets which is selected according at least partially to one or more confidence scores of the plurality of image offsets, wherein the matched points of interest correspond to a second telestration mark; and cause the first and second telestration marks to be displayed respectively over the first and second two-dimensional images on the stereo display such that the first telestration mark and the second telestration mark appear as a single three-dimensional telestration mark on the stereo display.

2. The medical system of claim 1 , further comprising:
a user interface that is usable by a first user to define the first telestration by indicating the user selected points of interest of the first two-dimensional image.

3. The medical system of claim 2 , wherein the user interface includes a touchscreen display for displaying the first two-dimensional image.

4. The medical system of claim 2 , wherein the user interface includes a monovision display and an input device, wherein the input device is usable by the first user to define the first telestration by indicating the user selected points of interest on the first two-dimensional image being displayed on the monovision display.

5. The medical system of claim 1 , wherein the plurality of image offsets includes at least two of: a global offset which the processor is programmed to compute by using a coarse-to-fine global offset image matching method, a region offset which the processor is programmed to compute by using a coarse-to-fine region image matching method, and a feature offset which the processor is programmed to compute by using a point matching method based upon feature detection and matching.

6. The medical system of claim 5 , wherein the processor is programmed to compute the global offset by using normalized cross correlations to compare information of the first and second two-dimensional images.

7. The medical system of claim 5 , wherein the processor is programmed to compute the region offset by matching a region of the first two-dimensional image to a region of the second two-dimensional image by using at least one of cross-correlation, two-way matching, least squares regression, and non-linear regression.

8. The medical system of claim 5 , wherein the processor is programmed to compute the feature offset by matching features of the first two-dimensional image to features of the second two-dimensional image by using at least one of Harris corner detection, scale-space extrema detection, local extrema detection, and scale invariant feature transform.

9. The medical system of claim 1 , wherein the processor is programmed to perform the selectively matching by at least one of selectively matching regions, selectively matching features, selectively interpolating features, and selectively interpolating previously matched points of interest.

10. The medical system of claim 9 , wherein the processor is programmed to determine region match confidence scores when the regions are selectively matched, determine feature match scores when the features are selectively matched, selectively match the features in response to the region match confidence scores, selectively interpolate the features in response to the feature match scores, and selectively interpolate the previously matched points of interest in response to the feature match scores; and wherein locations of the matched points of interest are determined by using the region match confidence scores and the feature match scores.

Assignees

• Intuitive Surgical Operations

Inventors

• Zhao Wenyi
• Wu Chenyu
• Hirvonen David
• Hasser Christopher J
• Miller Brian E
• Mohr Catherine J
• Zhao Tao
• Di Maio Simon
• Hoffman Brian D

Classifications

• H04N13/10

  Processing, recording or transmission of stereoscopic or multi-view image signals · CPC title

• A61B2034/2065

  Tracking using image or pattern recognition · CPC title

• A61B34/20

  Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis · CPC title

• H04N2013/0092

  Image segmentation from stereoscopic image signals · CPC title

• A61B2034/2057

  Details of tracking cameras · CPC title