Portable electronic equipment and method of controlling a portable electronic equipment

Patent metadata

Field	Value
Publication number	US-10216267-B2
Application number	US-201414442591-A
Country	US
Kind code	B2
Filing date	May 8, 2014
Priority date	May 8, 2014
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

A portable electronic equipment comprises a gaze tracking device configured to track an eye gaze direction of a user in a time interval. The portable electronic equipment comprises a processing device coupled to the gaze tracking device and configured to generate heat map data for the eye gaze direction, generate saliency map data for a field of view of the user, and identify at least one point of interest in the field of view based on both the heat map data and the saliency map data.

First claim

The invention claimed is:

1. A portable electronic equipment, comprising:
a gaze tracking device configured to track an eye gaze direction of a user in a time interval;
a processing device coupled to the gaze tracking device and configured to
generate heat map data for the eye gaze direction, wherein the heat map data provides, for each one of a plurality of gaze points, information on the fraction of time within the time interval for which the user has been gazing towards this gaze point or towards an area around this gaze point;
generate saliency map data for a field of view of the user;
combine the heat map data with the saliency map data to provide a combined data map; and
identify at least one point of interest in the field of view based on the combined data map.

2. The portable electronic equipment of claim 1 , further comprising:
an image capturing device configured to capture at least one image of the field of view of the user,
wherein the processing device is configured to process the at least one image of the field of view to generate the saliency map data.

3. The portable electronic equipment of claim 2 ,
wherein the processing device is configured to control the image capturing device based on the identified at least one point of interest.

4. The portable electronic equipment of claim 3 ,
wherein the processing device is configured to set a focal point of the image capturing device based on the identified at least one point of interest.

5. The portable electronic equipment of claim 2 ,
wherein the gaze tracking device comprises an image sensor which has a lower resolution than the image capturing device.

6. The portable electronic equipment of claim 2 , further comprising:
an optical output device configured to output the at least one image captured by the image capturing device,
wherein the processing device is configured to generate the heat map data for a gaze point on the optical output device.

7. The portable electronic equipment of claim 1 ,
wherein the portable electronic equipment comprises an optical output device, and
wherein the processing device is configured to
control the optical output device to output graphical elements in the field of view of the user, and
generate the saliency map data based on coordinates at which the graphical elements are output by the optical output device.

8. The portable electronic equipment of claim 7 ,
wherein the processing device is configured to determine, based on the heat map data and the saliency map data, a graphical element of the graphical elements which is the at least one point of interest.

9. The portable electronic equipment of claim 7 ,
wherein the graphical elements are selected from a group consisting of:
augmented reality elements;
graphical user interface elements; and
game graphics.

10. The portable electronic equipment of claim 1 ,
wherein the processing device is configured to process an output of the gaze tracking device to determine a gaze trajectory and to generate the heat map data from the gaze trajectory.

11. The portable electronic equipment of claim 1 ,
which is a handheld communication terminal.

12. The portable electronic equipment of claim 1 ,
which is a head-mounted device.

13. The portable electronic equipment of claim 1 , wherein the heat map data is indicative of a fraction of time the user spent gazing at various regions in a field of view.

14. A method of controlling a portable electronic equipment, comprising:
monitoring an eye gaze direction of a user in a time interval;
generating heat map data for the eye gaze direction, wherein the heat map data provides, for each one of a plurality of gaze points, information on the fraction of time within the time interval for which the user has been gazing towards this gaze point or towards an area around this gaze point;
generating saliency map data for a field of view of the user;
combining the heat map data with the saliency map data to provide a combined data map;
identifying at least one point of interest in the field of view based on the combined data map; and
controlling the portable electronic equipment in dependence on the identified at least one point of interest.

15. The method of claim 14 ,
wherein the portable electronic equipment comprises an image capturing device, and
wherein controlling the portable electronic equipment comprises:
adjusting settings of the image capturing device based on the identified at least one point of interest.

16. The method of claim 15 , which is performed by the portable electronic equipment
wherein controlling the portable electronic equipment comprises:
setting a focal point of the image capturing device based on the identified at least one point of interest.

17. The method of claim 15 ,
wherein the portable electronic equipment comprises a gaze tracking device which monitors the eye gaze direction of the user, the gaze tracking device comprising an image sensor which has a lower resolution than the image capturing device.

18. The method of claim 15 , further comprising:
outputting, by an optical output device of the portable electronic equipment, at least one image captured by the image capturing device,
wherein the heat map data is generated for a gaze point on the optical output device.

19. The method of claim 14 ,
wherein the portable electronic equipment comprises an optical output device,
wherein the method further comprises outputting graphical elements in the field of view of the user, and
wherein the saliency map data are generated based on coordinates at which the graphical elements are output by the optical output device.

20. The method of claim 19 ,
wherein identifying at least one point of interest comprises:
determining, based on the heat map data and the saliency map data, a graphical element of the graphical elements which is the at least one point of interest.

Assignees

• Sony Corp
• Sony Mobile Communications Inc

Inventors

• Thorn Ola
• Aronsson Par-Anders

Classifications

• G06T7/74

  involving reference images or patches · CPC title

• G06T2207/30241

  Trajectory · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06T2207/30201

  Face · CPC title