Terminal with line-of-sight tracking function, and method and apparatus for determining point of gaze of user

Patent metadata

Field	Value
Publication number	US-10929659-B2
Application number	US-201916281894-A
Country	US
Kind code	B2
Filing date	Feb 21, 2019
Priority date	Aug 22, 2016
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

A terminal with a line-of-sight tracking function is disclosed. The terminal with a line-of-sight tracking function includes a body, a camera, and at least two light emitting diodes. The camera and the at least two light emitting diodes are mounted on the body, so that the terminal can emit a ray by using the at least two light emitting diodes, to ensure that the emitted ray can be shined on an eye of the user when the user is at different angles. After the ray is reflected by the eye of the user, the terminal can collect the reflected ray by using the camera, obtain an eye image of the user, and track a line of sight of the user based on the eye image, thereby increasing a success rate of line-of-sight tracking.

First claim

What is claimed is:

1. A terminal with a line-of-sight tracking function, wherein the terminal comprises:
a body;
a camera mounted on the body and comprising a mask filter layer, wherein the mask filter layer comprises at least two filter units; and
at least two light emitting diodes (LEDs) mounted on the body and comprising a first LED, a second LED, and a third LED, wherein each of the LEDs is configured to emit a respective ray, and
wherein the camera is configured to collect a ray corresponding to a reflected emitted ray, wherein the first LED comprises a first frequency band that is the same as an operating frequency band of a filter unit of the filter units, wherein the second LED comprises a second frequency band that is the same as an operating frequency band of another filter unit of the filter units, wherein the filter units comprise still another filter unit, wherein the third LED comprises a third frequency band that is the same as an operating frequency band of the still other filter unit, and wherein the first frequency band is different than the second frequency band.

2. The terminal of claim 1 , wherein a distance between the first LED and the camera is shorter than a distance between the second LED and the camera.

3. The terminal of claim 2 , wherein the LEDs further comprise a third LED, wherein the third LED comprises a third frequency band that is different than both the first frequency band and the second frequency band, and wherein the distance between the first LED and the camera is shorter than a distance between the third LED and the camera.

4. The terminal of claim 1 , wherein the LEDs comprise a fourth LED, a fifth LED, a sixth LED, and a seventh LED, wherein a distance between the fourth LED and the camera is the shortest in distances between all the LEDs and the camera, and wherein a frequency band of the fourth LED is different than a frequency band of the fifth LED, a frequency band of the sixth LED, and a frequency band of the seventh LED.

5. The terminal of claim 4 , wherein the camera is in an upper left corner of the body, wherein the fourth LED is in the upper left corner of the body, wherein the fifth LED is in an upper right corner of the body, wherein the sixth LED is in a lower left corner of the body, and wherein the seventh LED is in a lower right corner of the body.

6. The terminal of claim 4 , wherein the frequency band of the fifth LED and the frequency band of the sixth LED are both the same as the frequency band of the seventh LED, wherein the frequency band of the fourth LED is the same as an operating frequency band of one filter unit in the filter units, and wherein the frequency band of the fifth LED is the same as an operating frequency band of a second filter unit in the filter units.

7. The terminal according to claim 4 , wherein the frequency band of the sixth LED is different than the frequency band of the fifth LED, wherein the frequency band of the fifth LED is the same as the frequency band of the seventh LED, the mask filter layer comprises, wherein the frequency band of the fourth LED is the same as an operating frequency band of the first filter unit, wherein the frequency band of the fifth LED is the same as an operating frequency band of the second filter unit, and wherein the frequency band of the sixth LED is the same as an operating frequency band of a third filter unit.

8. The terminal of claim 1 , wherein the LEDs are infrared emitting diodes, and wherein the camera is an infrared camera.

9. A method for determining a point of gaze of a user of a terminal, wherein the method comprises:
emitting a first ray by a first light source;
emitting a second ray by a second light source;
emitting a third ray by a third light source, wherein a frequency band of the first ray is different than a frequency band of the second ray, wherein the terminal comprises a camera comprising a mask filter layer, wherein the mask filter layer comprises filter units, wherein a first frequency band of the first light source is the same as an operating frequency band of a filter unit of filter units, and wherein a third frequency band of the second light source is the same as an operating frequency band of another filter unit in the filter units, wherein the filter units comprise still another filter unit, wherein the third frequency band is the same as an operating frequency band of the still other filter unit, and wherein the first light source, the second light source, and the camera are all located on the terminal;
obtaining, with the camera, a first image based on a first filter ray, wherein the first filter ray is obtained after a reflected first ray passes through the mask filter layer;
obtaining, with the camera, a second image based on a second filter ray, wherein the second filter ray is obtained after a reflected second ray passes through the mask filter layer; and
determining the point of gaze of the user based on the first image and the second image.

10. The method of claim 9 , wherein determining the point of gaze of the user based on the first image and the second image comprises:
determining pupil location information and reflection point location information of the user based on the first image and the second image; and
determining the point of gaze of the user based on the pupil location information, the reflection point location information, and a preset eyeball geometry model.

11. The method of claim 9 , wherein a moment of emitting the first ray by the first light source is the same as a moment of emitting the second ray by the second light source.

12. An apparatus for determining a point of gaze of a user of a terminal, wherein the apparatus comprises:
at least two light emitting diodes (LEDs), wherein the LEDs are configured to:
emit a first ray by a first LED;
emit a second ray by a second LED;
emit a third ray by a third LED;

at least one camera coupled to the LEDs, wherein the one camera comprises a mask filter layer comprising at least two filter units, and wherein the camera is configured to:
obtain a first filter ray based on a reflected first ray passing through the mask filter layer;
obtain a first image based on the first filter ray;
obtain a second filter ray based on a reflected second ray passing through the mask filter layer;
obtain a second image based on the second filter ray; and

a processor coupled to the camera and configured to determine the point of gaze of the user based on the first image and the second image, wherein the first LED comprises a first frequency band that is the same as an operating frequency band of a filter unit of the filter units, and wherein the second LED comprises a second frequency band that is the same as an operating frequency band of another filter unit in the filter units, wherein the third LED comprises a third frequency band that is the same as an operating frequency band of another filter unit of the filter units, wherein a frequency band of the first ray is different than a frequency band of the second ray, wherein a distance between the first LED and the camera is shorter than a distance between the second LED and the camera, and wherein the first LED, the second LED, and the camera are all located on the terminal.

13. The apparatus of claim 12 , wherein the processor is further configured to:
determine pupil location information and reflection point location information of the user based on the first image and the second image; and
determine the point of gaze of the user based on the pupil location information, the reflection point location information, and a preset eyeball geometry model.

14. The app

Assignees

• Huawei Tech Co Ltd

Inventors

• Jin Jiaoying
• Lin Tao
• Liu Yingchun

Classifications

• H04N23/56

  provided with illuminating means · CPC title

• H04B10/502 Primary

  LED transmitters · CPC title

• G06V40/19 Primary

  Sensors therefor · CPC title

• G06F3/01

  Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title