Method and an apparatus for determining a gaze point on a three-dimensional object

Patent metadata

Field	Value
Publication number	US-10699439-B2
Application number	US-201916290934-A
Country	US
Kind code	B2
Filing date	Mar 3, 2019
Priority date	Sep 17, 2012
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

A system for determining the gaze endpoint of a subject, the system comprising: a eye tracking unit adapted to determine the gaze direction of one or more eyes of the subject; a head tracking unit adapted to determine the position comprising location and orientation of the eye tracker with respect to a reference coordinate system; a 3D Structure representation unit, that uses the 3D structure and position of objects of the scene in the reference coordinate system to provide a 3D structure representation of the scene; based on the gaze direction, the eye tracker position and the 3D structure representation, calculating the gaze endpoint on an object of the 3D structure representation of the scene or determining the object itself.

First claim

What is claimed is:

1. A method comprising:
obtaining aggregated gaze data including, for a first object of a plurality of objects in a scene, a first amount that a plurality of users have gazed at the first object from a plurality of different perspectives
and, for a second object of a plurality of objects in a scene, a second amount that a plurality of users have gazed at the second object from a plurality of different perspectives;
obtaining an image of the scene;
projecting the aggregated gaze data on the image of the scene to generate a visualization image; and
displaying the visualization image, wherein the first object is displayed in the visualization image in a first manner based on the first amount and the second object is displayed in the visualization image in a second manner based on the second amount.

2. The method of claim 1 , wherein the first amount includes a first number of the plurality of users that have gazed at the first object and the second amount includes a second number of the plurality of users that have gazed at the second object.

3. The method of claim 1 , wherein the first amount includes a first amount of time the plurality of users have gazed at the first object and the second amount includes a second amount of time the plurality of users have gazed at the second object.

4. The method of claim 1 , wherein displaying the visualization image includes displaying a reference image in which the first object is displayed in the reference image in the first manner based on the first amount and the second object is displayed in the reference image in the second manner based on the second amount.

5. The method of claim 1 , wherein displaying the visualization image includes:
displaying, at a first time, a first image of the scene from a first perspective in which the first object is displayed in the first image in the first manner based on the first amount and the second object is displayed in the first image in the second manner based on the second amount; and
displaying, at a second time, a second image of the scene from a second perspective different than the first perspective in which the first object is displayed in the second image in the first manner based on the first amount and the second object is displayed in the second image in the second manner based on the second amount.

6. The method of claim 1 , wherein the first object is displayed with a first color based on the first amount and the second object is displayed with a second color based on the second amount.

7. The method of claim 1 , wherein the first object is displayed with a first texture based on the first amount and the second object is displayed with a second texture based on the second amount.

8. A system comprising:
a processor to:
obtain aggregated gaze data including, for a first object of a plurality of objects in a scene, a first amount that a plurality of users have gazed at the first object from a plurality of different perspectives and, for a second object of a plurality of objects in a scene, a second amount that a plurality of users have gazed at the second object from a plurality of different perspectives;
obtain an image of the scene;
project the aggregated gaze data on the image of the scene to generate a visualization image

a display to:
display the visualization image, wherein the first object is displayed in the visualization image in a first manner based on the first amount and the second object is displayed in the visualization image in a second manner based on the second amount.

9. The system of claim 8 , wherein the first amount includes a first number of the plurality of users that have gazed at the first object and the second amount includes a second number of the plurality of users that have gazed at the second object.

10. The system of claim 8 , wherein the first amount includes a first amount of time the plurality of users have gazed at the first object and the second amount includes a second amount of time the plurality of users have gazed at the second object.

11. The system of claim 8 , wherein the display is to display the visualization image by displaying a reference image in which the first object is displayed in the reference image in the first manner based on the first amount and the second object is displayed in the reference image in the second manner based on the second amount.

12. The system of claim 8 , wherein the display is to display the visualization image by:
displaying, at a first time, a first image of the scene from a first perspective in which the first object is displayed in the first image in the first manner based on the first amount and the second object is displayed in the second image in the second manner based on the second amount; and
displaying, at a second time, a second image of the scene from a second perspective different than the first perspective in which the first object is displayed in the second image in the first manner based on the first amount and the second object is displayed in the second image in the second manner based on the second amount.

13. The system of claim 8 , wherein the first object is displayed with a first color based on the first amount and the second object is displayed with a second color based on the second amount.

14. The system of claim 8 , wherein the first object is displayed with a first texture based on the first amount and the second object is displayed with a second texture based on the second amount.

15. A non-transitory computer-readable medium having instructions encoded thereon which, when executed by a processor of a device including a display, cause the device to:
obtain aggregated gaze data including, for a first object of a plurality of objects in a scene, a first amount that a plurality of users have gazed at the first object from a plurality of different perspectives and, for a second object of a plurality of objects in a scene, a second amount that a plurality of users have gazed at the second object from a plurality of different perspectives;
obtain an image of the scene;
project the aggregated gaze data on the image of the scene to generate a visualization image; and
display, on the display, the visualization image, wherein the first object is displayed in the visualization image in a first manner based on the first amount and the second object is displayed in the visualization image in a second manner based on the second amount.

16. The non-transitory computer-readable medium of claim 15 , wherein the first amount includes a first number of the plurality of users that have gazed at the first object and the second amount includes a second number of the plurality of users that have gazed at the second object.

17. The non-transitory computer-readable medium of claim 15 , wherein the first amount includes a first amount of time the plurality of users have gazed at the first object and the second amount includes a second amount of time the plurality of users have gazed at the second object.

18. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed, cause the device to display the visualization image by displaying a reference image in which the first object is displayed in the reference image in the first manner based on the first amount and the second object is displayed in the reference image in the second manner based on the second amount.

19. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed, cause the device to display the vis

Assignees

• Apple Inc

Inventors

• Hoffmann Jan
• Sengelaub Tom
• Williams Denis

Classifications

• G06V40/19 Primary

  Sensors therefor · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• A61B3/113

  for determining or recording eye movement · CPC title

• G06T2207/30201

  Face · CPC title