Method and apparatus for tracking eye based on eye reconstruction

Patent metadata

Field	Value
Publication number	US-11989911-B2
Application number	US-202217740076-A
Country	US
Kind code	B2
Filing date	May 9, 2022
Priority date	Dec 10, 2019
Publication date	May 21, 2024
Grant date	May 21, 2024

Abstract

An eye reconstruction-based eye tracking method and apparatus are provided. The eye tracking method includes generating a reconstructed image by performing eye reconstruction with respect to an input image, determining a difference value between the input image and the reconstructed image, selecting one of the input image, the reconstructed image, and a replacement image as a target image based on the determined difference value, and performing eye tracking based on the target image.

First claim

What is claimed is:

1. An eye tracking method comprising:
generating a reconstructed image by performing eye reconstruction with respect to an input image by reducing a noise component in the input image;
selecting a sample image corresponding to the input image among sample images stored in a database, as a replacement image of the input image;
selecting one of the input image, the reconstructed image, and the replacement image as a target image; and
performing eye tracking based on the target image,
wherein the generating comprises generating the reconstructed image based on a portion having a high priority among principal component vectors corresponding to the input image, and
wherein each of the principal component vectors corresponds to an eigenface predetermined based on principal component analysis on various face images.

2. The eye tracking method of claim 1 , further comprising determining a difference value between at least one first pixel included in the input image and at least one second pixel included in the reconstructed image corresponding to the at least one first pixel, and
wherein the selecting comprises selecting one of the input image, the reconstructed image, and the replacement image as the target image based on the determined difference value.

3. The eye tracking method of claim 2 , wherein the selecting comprises:
selecting the input image as the target image based on the determined difference value being less than a first threshold;
selecting the reconstructed image as the target image based on the determined difference value being greater than the first threshold and less than a second threshold; and
selecting the replacement image as the target image based on the determined difference value being greater than the second threshold, and
wherein the second threshold is greater than the first threshold.

4. The eye tracking method of claim 2 , wherein the replacement image is different from the input image and the reconstructed image.

5. The eye tracking method of claim 2 , wherein the sample image has a highest similarity to the input image, among the sample images stored in the database, and the highest similarity is determined based on a comparison between feature points of the input image and corresponding feature points of each of the sample images.

6. The eye tracking method of claim 5 , wherein the feature points of the input image and the corresponding feature points of each of the sample images are each extracted from a region except for eyes.

7. The eye tracking method of claim 5 , wherein the sample images correspond to images having previously succeeded in eye tracking.

8. The eye tracking method of claim 1 , further comprising:
storing the input image as a sample image in the database based on the eye tracking being successful based on the input image or the reconstructed image.

9. The eye tracking method of claim 1 , further comprising performing eye detection with respect to the input image,
wherein the generating is performed based on the eye detection being successful with respect to the input image.

10. The eye tracking method of claim 1 , wherein, based on the replacement image being selected as the target image, the performing eye tracking comprises performing the eye tracking based on eye position information mapped to the replacement image.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the eye tracking method of claim 1 .

12. An mobile device comprising:
a camera configured to obtain an input image of a user;
a memory configured to store instructions; and
a processor configured to execute the instructions to:
generate a reconstructed image by performing eye reconstruction with respect to the input image by reducing a noise component in the input image;
select a sample image corresponding to the input image among sample images stored in a database, as a replacement image of the input image;
determine a target image by selecting one of the input image, the reconstructed image, and the replacement image; and
perform eye tracking on the target image,

wherein the processor is further configured to generate the reconstructed image based on a portion having a high priority among principal component vectors corresponding to the input image, and
wherein each of the principal component vectors corresponds to an eigenface predetermined based on principal component analysis on various face images.

13. An eye tracking apparatus comprising:
a memory configured to store instructions; and
a processor configured to execute the instructions to:
generate a reconstructed image by performing eye reconstruction with respect to an input image by reducing a noise component in the input image;
select a sample image corresponding to the input image among sample images stored in a database, as a replacement image of the input image;
determine a target image by selecting one of the input image, the reconstructed image, and the replacement image; and
perform eye tracking on the target image,

wherein the processor is further configured to:
determine a difference value between at least one first pixel included in the input image and at least one second pixel included in the reconstructed image corresponding to the at least one first pixel, and
select one of the input image, the reconstructed image, and the replacement image as the target image based on the determined difference value.

14. The eye tracking apparatus of claim 13 , wherein the processor is further configured to generate the reconstructed image based on a portion having a high priority among principal component vectors corresponding to the input image, and
wherein each of the principal component vectors corresponds to an eigenface predetermined based on principal component analysis on various face images.

15. The eye tracking apparatus of claim 13 , wherein the processor is further configured to:
select the input image as the target image based on the determined difference value being less than a first threshold,
select the reconstructed image as the target image based on the determined difference value being greater than the first threshold and less than a second threshold, and
select the replacement image as the target image based on the determined difference value being greater than the second threshold, and
wherein the second threshold is greater than the first threshold.

16. The eye tracking apparatus of claim 15 , wherein the sample image has a highest similarity to the input image, among the sample images stored in the database, and the highest similarity is determined based on a comparison between feature points of the input image and corresponding feature points of each of the sample images, and
wherein the feature points of the input image and the corresponding feature points of each of the sample images are each extracted from a region except for eyes.

17. The eye tracking apparatus of claim 13 , wherein, based on the replacement image being selected as the target image, the processor is further configured to perform the eye tracking based on eye position information mapped to the replacement image.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Heo Jingu
• Kang Dongwoo
• Nam Dong Kyung

Classifications

• G06T7/74 Primary

  involving reference images or patches · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/254

  involving subtraction of images · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title