Gaze point determination method and apparatus, electronic device, and computer storage medium

Patent metadata

Field	Value
Publication number	US-11295474-B2
Application number	US-201916729424-A
Country	US
Kind code	B2
Filing date	Dec 29, 2019
Priority date	Sep 28, 2018
Publication date	Apr 5, 2022
Grant date	Apr 5, 2022

Abstract

A gaze point determination method and apparatus, an electronic device, and a computer storage medium are provided. The method includes: obtaining two-dimensional coordinates of eye feature points of at least one eye of a face in an image, the eye feature points including an eyeball center area feature point; obtaining, in the preset three-dimensional coordinate system, three-dimensional coordinate of a corresponding eyeball center area feature point in a three-dimensional face model corresponding to the face in the image based on the obtained two-dimensional coordinate of the eyeball center area feature point; and obtaining a determination result for a position of a gaze point of the eye of the face in the image according to two-dimensional coordinates of feature points other than the eyeball center area feature point in the eye feature points and the three-dimensional coordinate of the eyeball center area feature point in the preset three-dimensional coordinate system.

First claim

The invention claimed is:

1. A gaze point determination method, comprising:
obtaining two-dimensional coordinates of eye feature points of at least one eye of a face in an image, wherein the eye feature points comprise an eyeball center area feature point;
obtaining, in a preset three-dimensional coordinate system, a three-dimensional coordinate of a corresponding eyeball center area feature point in a three-dimensional face model corresponding to the face in the image based on the obtained two-dimensional coordinate of the eyeball center area feature point; and
obtaining a gaze fraction of the eye of the face in the image according to two-dimensional coordinates of feature points other than the eyeball center area feature point in the eye feature points and the three-dimensional coordinate of the eyeball center area feature point in the preset three-dimensional coordinate system, comparing the gaze fraction with a preset threshold to obtain the determination result for the position of the gaze point of the eye of the face in the image, and obtaining a determination result for a position of a gaze point of an eye of the face in the image,
wherein the comparing the gaze fraction with a preset threshold to obtain the determination result for the position of the gaze point of the eye of the face in the image comprises:
responsive to that the gaze fraction is greater than the preset threshold, determining that the gaze point of the eye of the face in the image is within a preset area; and
responsive to that the gaze fraction is less than or equal to the preset threshold, obtaining that the gaze point of the eye of the face in the image is out of the preset area,
wherein the method further comprises:
displaying the image according to a first preset display mode in response to that the gaze point of the eye of the face in the image is within the preset area; and
displaying the image according to a second preset display mode in response to that the gaze point of the eye of the face in the image is out of the preset area, wherein the first preset display mode and the second preset display mode respectively indicate setting frames of different colors for the image displayed.

2. The method according to claim 1 , wherein the eye feature points further comprise: an eyelid line feature point and an eyeball contour feature point.

3. The method according to claim 1 , wherein the obtaining two-dimensional coordinates of eye feature points of at least one eye of a face in an image comprises:
extracting features from the image, and obtaining two-dimensional coordinates of face feature points in the image; and
obtaining the two-dimensional coordinates of the eye feature points of at least one eye of the face in the image based on the two-dimensional coordinates of the face feature points in the image.

4. The method according to claim 3 , wherein the obtaining the two-dimensional coordinates of the eye feature points of at least one eye of the face in the image based on the two-dimensional coordinates of the face feature points in the image comprises:
intercepting one or more rectangular images comprising an eye area corresponding to the at least one eye from the image according to two-dimensional coordinates of feature points of eye area of the face in the image; and
extracting features from the rectangular image, and obtaining the two-dimensional coordinates of the eye feature points of at least one eye of the face in the image.

5. The method according to claim 4 , wherein the intercepting rectangular images comprising an eye area corresponding to the at least one eye from the image according to two-dimensional coordinates of feature points of the eye area of the face in the image comprises:
intercepting a rectangular image comprising an eye area corresponding to one eye from the image according to two-dimensional coordinates of feature points of the eye area of one eye of the face in the image; and
the extracting features from the rectangular image, and obtaining the two-dimensional coordinates of the eye feature points of at least one eye of the face in the image comprises:
extracting features from the rectangular image, and obtaining the two-dimensional coordinates of the eye feature points of one eye of the face in the image;
or,
mirroring the rectangular image, extracting features from the rectangular image and the mirrored rectangular image, and obtaining the two-dimensional coordinates of the eye feature points of two eyes of the face in the image.

6. The method according to claim 1 , wherein the preset area comprises: part or all of a screen area.

7. The method according to claim 1 , wherein the obtaining a gaze fraction of the eye of the face in the image according to the two-dimensional coordinates of the feature points other than the eyeball center area feature point in the eye feature points and the three-dimensional coordinate of the eyeball center area feature point in the preset three-dimensional coordinate system comprises:
obtaining, through a neural network, the gaze fraction of the eye of the face in the image according to the two-dimensional coordinates of the feature points other than the eyeball center area feature point in the eye feature points and the three-dimensional coordinate of the eyeball center area feature point in the preset three-dimensional coordinate system.

8. The method according to claim 7 , wherein after the obtaining, in the preset three-dimensional coordinate system, the three-dimensional coordinate of an eyeball center area feature point in a three-dimensional face model corresponding to the face in the image based on the obtained two-dimensional coordinate of the eyeball center area feature point, the method further comprises:
adjusting, according to a preset format, formats of the two-dimensional coordinates of the feature points other than the eyeball center area feature point in the eye feature points and the three-dimensional coordinate of the eyeball center area feature point in the preset three-dimensional coordinate system; and
the obtaining, through a neural network, the gaze fraction of the eye of the face in the image according to the two-dimensional coordinates of the feature points other than the eyeball center area feature point in the eye feature points and the three-dimensional coordinate of the eyeball center area feature point in the preset three-dimensional coordinate system comprises:
obtaining, through the neural network, the gaze fraction of the eye of the face in the image according to the two-dimensional coordinates of the feature points other than the eyeball center area feature point in the eye feature points and the three-dimensional coordinate of the eyeball center area feature point in the preset three-dimensional coordinate system which are subjected to format adjustment.

9. The method according to claim 1 , wherein the method is applied to at least one of the following: facial expression recognition, terminal payment, terminal locking, or terminal unlocking.

10. The method according to claim 1 , wherein the obtaining, in the preset three-dimensional coordinate system, the three-dimensional coordinate of a corresponding eyeball center area feature point in a three-dimensional face model corresponding to the face in the image based on the obtained two-dimensional coordinate of the eyeball center area feature point comprises:
generating the three-dimensional face model corresponding to the face in the image according to the face in the image; and
obtaining, in the preset three-dimensional coordinate system, the three-dimensional coordinate of the corresponding eyeball center area feature point in the three-dimensional face model based on the ob

Assignees

• Beijing Sensetime Tech Development Co Ltd

Inventors

• Liu Tinghao
• Wang Quan
• Qian Chen

Classifications

• G06V40/18 Primary

  Eye characteristics, e.g. of the iris · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/75 Primary

  involving models · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title