Method and an apparatus for determining a gaze point on a three-dimensional object

Patent metadata

Field	Value
Publication number	US-11676302-B2
Application number	US-202016878932-A
Country	US
Kind code	B2
Filing date	May 20, 2020
Priority date	Sep 17, 2012
Publication date	Jun 13, 2023
Grant date	Jun 13, 2023

Abstract

A system for determining the gaze endpoint of a subject, the system comprising: a eye tracking unit adapted to determine the gaze direction of one or more eyes of the subject; a head tracking unit adapted to determine the position comprising location and orientation of the eye tracker with respect to a reference coordinate system; a 3D Structure representation unit, that uses the 3D structure and position of objects of the scene in the reference coordinate system to provide a 3D structure representation of the scene; based on the gaze direction, the eye tracker position and the 3D structure representation, calculating the gaze endpoint on an object of the 3D structure representation of the scene or determining the object itself.

First claim

What is claimed is:

1. A method comprising:
determining gaze directions of two eyes of a subject via coordinates in a 3D reference coordinate system;
determining an intersection of the gaze directions of the two eyes based on a vergence of the two eyes upon determining the gaze directions converge;
choosing a point lying closest to the gaze directions as the intersection upon determining the gaze directions do not intersect in space;
determining a gaze endpoint in the 3D reference coordinate system based on the intersection; and
determining which object of the plurality of objects the subject is gazing at based on the gaze endpoint.

2. The method of claim 1 , further comprising:
in response to determining the object the subject is gazing at, displaying a tag associated with the object the subject is gazing at.

3. The method of claim 2 , wherein the tag is received from a second subject other than the subject.

4. The method of claim 1 , further comprising:
in response to determining the object the subject is gazing at, receiving, from the subject a tag; and
associating the tag with the object the subject is gazing at.

5. The method of claim 1 , further comprising:
in response to determining the object the subject is gazing at, highlighting the object the subject is gazing at in an image of the scene.

6. The method of claim 1 , wherein the gaze endpoint lies in an empty space where no object of the plurality of objects is located.

7. The method of claim 6 , wherein the object the subject is gazing at is determined by choosing the object of the plurality of objects having a smallest respective distance between the gaze endpoint and the object, and wherein choosing the object having the smallest respective distance is based on the vergence.

8. The method of claim 1 , wherein the gaze endpoint lies behind an object of the plurality of objects other than the object the subject is gazing at.

9. The method of claim 1 , further comprising determining a location on the object the subject is gazing at based on the gaze endpoint.

10. The method of claim 1 , further comprising determining a probability distribution of the gaze endpoint and determining the respective probability of one or more of the plurality of objects being the object the subject is gazing at based on the probability distribution of the gaze endpoint.

11. A non-transitory computer-readable medium having instructions encoded thereon which, when executed by one or more processors of an electronic device, cause the electronic device to:
determine gaze directions of two eyes of a subject;
determine an intersection of the gaze directions of the two eyes based on a vergence of the two eyes upon determining the gaze directions converge;
choose a point lying closest to the gaze directions as the intersection upon determining the gaze directions do not intersect in space;
represent a plurality of objects of a scene through their 3D position and/or structure via coordinates in a 3D reference coordinate system;
determine a gaze endpoint in the 3D reference coordinate system based on the intersection; and
determine which object of the plurality of objects the subject is gazing at based on the gaze endpoint.

12. The non-transitory computer-readable medium of claim 11 , wherein the instructions, when executed, further cause the electronic device to:
in response to determining the object the subject is gazing at, display a tag associated with the object the subject is gazing at.

13. The non-transitory computer-readable medium of claim 11 , wherein the instructions, when executed, further cause the electronic device to:
in response to determining the object the subject is gazing at, receive, from the subject, a tag; and
associate the tag with the object the subject is gazing at.

14. The non-transitory computer-readable medium of claim 11 , wherein the instructions, when executed, further cause the electronic device to:
in response to determining the object the subject is gazing at, highlight object subject is gazing at in an image of the scene.

15. The non-transitory computer-readable medium of claim 11 , wherein the gaze endpoint lies in an empty space where no object of the plurality of objects is located.

16. The non-transitory computer-readable medium of claim 11 , wherein the gaze endpoint lies behind an object of the plurality of objects other than the object the subject is gazing at.

17. An electronic device comprising:
an eye tracker to determine gaze directions of two eyes of a subject;
a memory to store a representation of a plurality of objects of a scene through their 3D position and/or structure via coordinates in a 3D reference coordinate system; and
one or more processors to:
determine an intersection of the gaze directions of the two eyes based on a vergence of the two eyes upon determining the gaze directions converge;
choose a point lying closest to the gaze directions as the intersection upon determining the gaze directions do not intersect in space; and
determine a gaze endpoint in the 3D reference coordinate system based on the intersection, and determine which object of the plurality of objects the subject is gazing at based on the gaze endpoint.

18. The electronic device of claim 17 , further comprising:
a display to, in response to the one or more processors determining the object the subject is gazing at, display a tag associated with the object the subject is gazing at.

19. The electronic device of claim 17 , further comprising:
an input device to, in response to the one or more processors determining the object the subject is gazing at, receive, from the subject, a tag, wherein the one or more processors associate the tag the object the subject is gazing at.

20. The electronic device of claim 17 , further comprising:
a display to, in response to the one or more processors determining the object the subject is gazing at, display an image of the scene with the object the subject is gazing at highlighted.

21. The non-transitory computer-readable medium of claim 12 . wherein the tag is received from a second subject other than the subject.

22. The non-transitory computer-readable medium of claim 15 , wherein the object the subject is gazing at is determined by choosing the object of the plurality of objects having a smallest respective distance between the gaze endpoint and the object, and wherein choosing the object having the smallest respective distance is based on the vergence.

23. The non-transitory computer-readable medium of claim 11 , wherein the instructions, when executed, further cause the electronic device to determine a location on the object the subject is gazing at based on the gaze endpoint.

24. The non-transitory computer-readable medium of claim 11 , wherein the instructions, when executed, further cause the electronic device to determine a probability distribution of the gaze endpoint and determine the respective probability of one or more of the plurality of objects being the object the subject is gazing at based on the probability distribution of the gaze endpoint.

25. The electronic device of claim 18 , wherein the tag is received from a second subject other than the subject.

26. The electronic device of claim 17 , wherein the gaze endpoint lies in an empty space where no object of the plurality of objects is located.

27. The electronic device of cl

Assignees

• Apple Inc

Inventors

• Hoffmann Jan
• Sengelaub Tom
• Williams Denis

Classifications

• G06T7/75 Primary

  involving models · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• A61B3/113

  for determining or recording eye movement · CPC title

• G06T2200/04

  involving 3D image data · CPC title