Method and system for stereo gaze tracking

Patent metadata

Field	Value
Publication number	US-9503713-B2
Application number	US-201213662653-A
Country	US
Kind code	B2
Filing date	Oct 29, 2012
Priority date	Nov 2, 2011
Publication date	Nov 22, 2016
Grant date	Nov 22, 2016

Abstract

Stereo gaze tracking estimates a 3-D gaze point by projecting determined right and left eye gaze points on left and right stereo images. The determined right and left eye gaze points are based on one or more tracked eye gaze points, estimates for non-tracked eye gaze points based upon the tracked gaze points and image matching in the left and right stereo images, and confidence scores indicative of the reliability of the tracked gaze points and/or the image matching.

First claim

What is claimed is:

1. A method for stereo gaze tracking, the method comprising:
tracking, with a tracking system having a single-eye tracker, a gaze point of a first eye of a user on a first one of right and left two-dimensional display screens of a stereo viewer, wherein the right and left two-dimensional display screens respectively display information of right and left stereo images of a scene captured by a stereoscopic camera, wherein the right and left two-dimensional display screens are visible to the user through right and left eyepieces, respectively; and
estimating, by a processor, a gaze point of a second eye of the user on a second one of the right and left two-dimensional display screens by processing information, received from the tracking system, describing the tracked gaze point of the first eye only and information of the right and left stereo images of the scene.

2. The method of claim 1 , further comprising:
estimating a three-dimensional gaze point using the tracked gaze point of the first eye and the estimated gaze point of the second eye of the user.

3. A method for stereo gaze tracking of a user, the method comprising:
tracking, with a tracking system, a gaze point of a first eye of the user on a first one of right and left two-dimensional display screens of a stereo viewer, wherein the right and left two-dimensional display screens respectively display information of right and left stereo images of a scene captured by a stereoscopic camera;
estimating, by a processor, a gaze point of a second eye of a user on a second one of the right and left two-dimensional display screens by processing information of the tracked gaze point of the first eye only and information of the right and left stereo images of the scene, wherein the processing of the information of the tracked gaze point of the first eye and the information of the right and left stereo images of the scene comprises:
determining one or more of a global offset of the right and left stereo images, a region offset of corresponding regions of the right and left stereo images using the global offset and the tracked gaze point of the first eye, or a feature offset of corresponding features within the corresponding regions of the right and left stereo images; and
estimating the gaze point of the second eye of the user on the second one of the right and left two-dimensional display screens by using one of the region offset, the feature offset, an interpolation of previously determined and prior-in-time gaze points of the second eye of the user, or the global offset according to a reliability of at least one of the determinations and an availability of the previously determined and prior-in-time gaze points of the second eye of the user.

4. The method of claim 3 , wherein the determining of the global offset of the right and left stereo images comprises:
determining a left pyramid for the left stereo image;
determining a right pyramid for the right stereo image; and
determining the global offset using a multi-level process based on normalized cross correlations of corresponding levels of the left and right pyramids.

5. The method of claim 3 , wherein the determining of the region offset of corresponding regions of the right and left stereo images comprises:
determining a region in the first one of the right and left two-dimensional display screens using the tracked gaze point of the first eye;
determining a corresponding region in the second one of the right and left two-dimensional display screens using the tracked gaze point of the first eye and the global offset; and
determining a region offset by comparison of the regions in the first and second ones of the right and left two-dimensional display screens.

6. The method of claim 3 , wherein the determining of the feature offset of corresponding features of the right and left stereo images comprises:
identifying features proximal to the tracked gaze point of the first eye in the determined region in the first one of the right and left two-dimensional display screens;
identifying corresponding features in the determined corresponding region in the second one of the right and left two-dimensional display screens; and
determining a feature offset indicative of the gaze point of the second eye on the second one of the right and left two-dimensional display screens using the locations of the identified corresponding features.

7. The method of claim 3 , wherein the estimating of the gaze point of the second eye of the user on the second one of the right and left two-dimensional display screens includes determining which one of the region offset, the feature offset, the interpolation of previously determined and prior-in-time gaze points of the second eye of the user, and the global offset is to be used for the estimation by:
determining a first confidence score indicative of a reliability of the region offset determination;
determining whether the first confidence score is greater than a first threshold value; and
if the first confidence score is determined to be greater than the first threshold value, then using the region offset, otherwise, using one of the feature offset, the interpolation of previously determined and prior-in-time gaze points of the second eye of the user, and the global offset.

8. The method of claim 7 , wherein the determining of which one of the region offset, the feature offset, the interpolation of previously determined and prior-in-time gaze points of the second eye of the user, and the global offset is to be used for the estimation further comprises:
if the first confidence score is determined not to be greater than the first threshold value, then determining whether the first confidence score is greater than a second threshold value which is less than the first threshold value;
if the first confidence score is determined to be greater than the second threshold value, then using one of the feature offset, the interpolation of previously determined and prior-in-time gaze points of the second eye of the user, and the global offset; and
if the first confidence score is determined not to be greater than the second threshold value, then using one of the interpolation of previously determined and prior-in-time gaze points of the second eye of the user, and the global offset according to the availability and a reliability of the previously determined and prior-in-time gaze points of the second eye of the user.

9. The method of claim 8 , wherein if the first confidence score is determined not to be greater than the first threshold value, but greater than the second threshold value, then using the feature offset to estimate the gaze point of the second eye of the user on the second one of the right and left two-dimensional display screens.

10. The method of claim 8 , wherein if the first confidence score is determined not to be greater than the first threshold value, but greater than the second threshold value, then
determining a second confidence score indicative of a reliability of the feature offset determination,
determining whether the second confidence score is greater than the second threshold value,
if the second confidence score is determined to be greater than the second threshold value, then using the feature offset, and
if the second confidence score is determined not to be greater than the second threshold value, then using one of the interpolation of previously determined and prior-in-time gaze points of the second eye of the user, and the global offset according to the availability and the reliability of the previously determined and prior-in-time gaze points of the second eye of the user.

11. The method of claim 3 ,

Assignees

• Intuitive Surgical Operations
• Imp Innovations Ltd

Inventors

• Zhao Wenyi
• Itkowitz Brandon D
• Mylonas George
• Yang Gung-Zhong

Classifications

• H04N13/0484 Primary

  Electricity · mapped topic

• G06K9/00604

  Physics · mapped topic

• A61B3/113

  for determining or recording eye movement · CPC title

• A61B3/0058

  for multiple images · CPC title

• G06V40/19

  Sensors therefor · CPC title