Gaze direction mapping

Patent metadata

Field	Value
Publication number	US-10228763-B2
Application number	US-201816129585-A
Country	US
Kind code	B2
Filing date	Sep 12, 2018
Priority date	Aug 7, 2015
Publication date	Mar 12, 2019
Grant date	Mar 12, 2019

Abstract

A method for determining correspondence between a gaze direction and an environment around a wearable device is disclosed. The wearable device may include an eye tracking device and an outward facing image sensor. The method may include receiving an input parameter and at least one scene image from the outward facing image sensor. The method may further include determining, with at least the eye tracking device, at least one gaze direction of a wearer of the wearable device at a point in time corresponding to when the scene image was captured by the outward facing image sensor. The method may additionally include determining, based at least in part on the input parameter, that a particular scene image includes at least a portion of a predefined image. The method may moreover include determining, based on the at least one gaze direction, at least one gaze point on the particular scene image.

First claim

What is claimed is:

1. A method for determining a correspondence between a gaze direction and an environment around a wearable device, wherein the wearable device comprises an eye tracking device and an outward facing image sensor, and wherein the method comprises:
receiving at least one scene image from the outward facing image sensor;
determining, with at least the eye tracking device, a gaze direction of a wearer of the wearable device at a point in time corresponding to when the scene image was captured by the outward facing image sensor;
determining that a particular scene image includes at least a portion of a predefined image;
determining, based on the gaze direction, a first location of a gaze point on the particular scene image;
determining, based on the gaze direction, a second location of the gaze point on the portion of the predefined image;
extracting a first image subset including a first area of the particular scene image around the first location;
extracting a second image subset including a second area of the portion of the predefined image around the second location;
determining a similarity between the first image subset and the second image subset;
determining, based on the similarity, a confidence value for the gaze point, the confidence value representing a probability that the gaze point relates to the predefined image;
causing the particular scene image to be displayed with an overlaid visual indication of the gaze point, wherein the overlaid visual indication is based, in part, on the confidence value.

2. The method of claim 1 , further comprising determining a quality value for the gaze point, the quality value representing a degree of error in the gaze point on the particular scene image,
wherein the determining the first area associated with the gaze point is further based on the quality value.

3. The method of claim 2 , further comprising comparing the confidence value and the quality level,
wherein the determining the first area is further based on determining based on the comparison, that the confidence value is high and the quality value is low.

4. The method of claim 1 , wherein the confidence value indicates characteristics associated with the gaze point.

5. The method of claim 1 , further comprising:
determining an additional gaze direction at an additional point in time corresponding to when an additional scene image was captured by the outward facing image sensor;
determining that the additional scene image includes at least an additional portion of the predefined image;
determining, based on the additional gaze direction, an additional gaze point on the additional scene image;
determining an additional confidence value for the additional gaze point; and
based on the additional confidence value, providing an indication that the additional gaze direction is to be manually analyzed.

6. The method of claim 1 , further comprising:
determining a first feature of the predefined image;
determining a second feature of the particular scene image;
matching the first feature to the second feature; and
triangulating 3D points based on the matching the first feature to the second feature,
generating a 3D model of the predefined image based on the triangulated 3D points.

7. The method of claim 1 , further comprising:
receiving data indicating an event in the environment around the wearable device;
responsive to receiving the event data, modifying the confidence value.

8. The method of claim 7 , wherein the modified confidence value is lower than the confidence value.

9. A system for determining a correspondence between a gaze direction and an environment around a wearable device, wherein the system comprises:
the wearable device, comprising an eye tracking device and an outward facing image sensor, and
one or more processors configured to perform operations comprising:
receiving at least one scene image from the outward facing image sensor;
determining, with at least the eye tracking device, a gaze direction of a wearer of the wearable device at a point in time corresponding to when the scene image was captured by the outward facing image sensor;
determining that a particular scene image includes at least a portion of a predefined image;
determining, based on the gaze direction, a first location of a gaze point on the particular scene image;
determining, based on the gaze direction, a second location of the gaze point on the portion of the predefined image;
extracting a first image subset including a first area of the particular scene image around the first location;
extracting a second image subset including a second area of the portion of the predefined image around the second location;
determining a similarity between the first image subset and the second image subset;
determining, based on the similarity, a confidence value for the gaze point, the confidence value representing a probability that the gaze point relates to the predefined image;
causing the particular scene image to be displayed with an overlaid visual indication of the gaze point, wherein the overlaid visual indication is based, in part, on the confidence value.

10. The system of claim 9 , further comprising determining a quality value for the gaze point, the quality value representing a degree of error in the gaze point on the particular scene image,
wherein the determining the first area associated with the gaze point is further based on the quality value.

11. The system of claim 10 , further comprising comparing the confidence value and the quality level,
wherein the determining the first area is further based on determining based on the comparison, that the confidence value is high and the quality value is low.

12. The system of claim 9 , wherein the confidence value indicates characteristics associated with the gaze point.

13. The system of claim 9 , further comprising:
determining an additional gaze direction at an additional point in time corresponding to when an additional scene image was captured by the outward facing image sensor;
determining that the additional scene image includes at least an additional portion of the predefined image;
determining, based on the additional gaze direction, an additional gaze point on the additional scene image;
determining an additional confidence value for the additional gaze point; and
based on the additional confidence value, providing an indication that the additional gaze direction is to be manually analyzed.

14. The system of claim 9 , further comprising:
determining a first feature of the predefined image;
determining a second feature of the particular scene image;
matching the first feature to the second feature; and
triangulating 3D points based on the matching the first feature to the second feature,
generating a 3D model of the predefined image based on the triangulated 3D points.

15. The system of claim 9 , further comprising:
receiving data indicating an event in the environment around the wearable device;
responsive to receiving the event data, modifying the confidence value.

16. The system of claim 15 , wherein the modified confidence value is lower than the confidence value.

17. A non-transitory machine readable medium having instructions stored thereon for determining a correspondence between a gaze direction and an environment around a wearable device, the wearable device comprising an eye tracking device and an outward facing image sensor, wherein the instructions are executable by one or more processors for at least:
receiving at least one s

Assignees

• Tobii Ab

Inventors

• Algotsson André
• Clausen Anders
• Högström Jesper
• Högström Jonas
• Lindgren Tobias
• Petersson Rasmus
• Skogö Mårten
• Wong Wilkey

Classifications

• G06F17/30047

  Physics · mapped topic

• G02B27/017

  Head mounted · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title