Image processing method and system for eye-gaze correction

Patent metadata

Field	Value
Publication number	US-10602077-B2
Application number	US-201715838364-A
Country	US
Kind code	B2
Filing date	Dec 12, 2017
Priority date	Oct 11, 2017
Publication date	Mar 24, 2020
Grant date	Mar 24, 2020

Abstract

An image processing method and an image processing system for eye-gaze correction are provided. The method is applicable to an image processing system having a screen and an image capturing device and includes the following steps. A user's face in front of the screen is captured by the image capturing device to generate a facial image. A head offset and an eye-gaze position of the user with respect to the screen are obtained based on the facial image so as to accordingly determine whether to correct the facial image. If yes, the facial image is corrected based on the eye-gaze position and a preset codebook to generate a corrected facial image, where the preset codebook records correction information of multiple preset eye-gaze positions.

First claim

What is claimed is:

1. An image processing method for eye-gaze correction, applicable to an image processing system having a screen and an image capturing device, the image processing method comprising:
capturing a face of a user in front of the screen by using the image capturing device to generate a facial image;
obtaining a head offset and an eye-gaze position of the user with respect to the screen from the facial image, wherein the head offset is calculated according to a configuration position of the image capturing device and a size of the screen;
determining whether to correct the facial image according to the head offset and the eye-gaze position; and
in response to the facial image determined to be corrected, correcting the facial image based on the eye-gaze position and a preset codebook to generate a corrected facial image, wherein the preset codebook records correction information of a plurality of preset eye-gaze positions.

2. The method according to claim 1 , wherein the step of obtaining the head offset and the eye-gaze position of the user with respect to the screen from the facial image comprises:
calculating a vertical distance and an orientation of the user with respect to the screen according to an area of a facial region in the facial image, the configuration position of the image capturing device, and the size of the screen so as to obtain the head offset.

3. The method according to claim 2 , wherein a first correspondence relationship between distances and areas is prestored in the image processing system, wherein the first correspondence relationship records average facial areas with respect to different preset vertical distances, and wherein the step of calculating the vertical distance and the orientation of the user with respect to the screen according to the area of the facial region in the facial image, the configuration position of the image capturing device, and the size of the screen comprises:
obtaining the area of the facial region in the facial image, obtaining a distance corresponding to the area of the facial region according to the first correspondence relationship, and setting the obtained distance as the vertical distance of the user with respect to the screen; and
obtaining a ratio of distances between the facial region to two sides of the screen to accordingly calculate the orientation of the user with respect to the screen.

4. The method according to claim 2 , wherein the vertical distance and the orientation of the user with respect to the screen is a vertical distance and an orientation of the user with respect to a center of the screen.

5. The method according to claim 2 , wherein the vertical distance and the orientation of the user with respect to the screen is a vertical distance and an orientation of the user with respect to a center of a displayed image on the screen.

6. The method according to claim 1 , wherein the step of obtaining the head offset and the eye-gaze position of the user with respect to the screen from the facial image comprises:
setting a plurality of gaze label points in a fixed region of the facial image; and
determining the gaze label point corresponding to an eye-gaze of the user by leveraging a machine learning algorithm and setting the determined gaze label point as the eye-gaze position.

7. The method according to claim 1 , wherein the step of determining whether to correct the facial image according to the head offset and the eye-gaze position comprises:
determining whether the head offset is greater than a preset offset;
in response to the head offset greater than the preset offset, not correcting the facial image; and
in response to the head offset not greater than the preset offset, determining whether the eye-gaze position is within a preset region so as to accordingly determine whether to correct the facial image.

8. The method according to claim 7 , wherein the preset region is at an eye region in the facial image, and wherein the step of determining whether the eye-gaze position is within the preset region so as to accordingly determine whether to correct the facial image comprises:
determining whether the eye-gaze position is within the preset region of the facial image;
in response to the eye-gaze position not within the preset region of the facial image, not correcting the facial image; and
in response to the eye-gaze position within the preset region of the facial image, correcting the facial image.

9. The method according to claim 6 , wherein the correction information recorded in the preset codebook is a correction vector of each of the preset eye-gaze positions, and wherein the step of correcting the facial image based on the eye-gaze position and the preset codebook to generate the corrected facial image comprises:
obtaining an angle between the eye-gaze position and a center of the fixed region and setting the obtained angle as an eye-gaze correction angle;
looking up the preset codebook by using the eye-gaze correction angle to obtain the correction vector; and
generating the corrected facial image according to the corrected vector and the facial age.

10. The method according to claim 1 , wherein after the step of generating the corrected facial image, the method further comprises a step of:
performing image smoothing on the corrected facial image.

11. The method according to claim 1 , wherein the image processing system is a video conference system further comprising a communication interface, and wherein after the step of generating the corrected facial image, the method further comprises a step of:
transmitting the corrected facial image to another electronic device via the communication interface.

12. The method according to claim 1 , wherein the image processing system is a self-portrait system, and wherein after the step of generating the corrected facial image, the method further comprises a step of:
displaying the corrected facial image on the screen.

13. An image processing system comprising:
a screen;
an image capturing device, disposed on a same side as the screen, and configured to capture a face of a user in front of the screen to generate a facial image;
a memory, configured to store data, images, and a preset codebook, wherein the preset codebook records correction information of a plurality of preset eye-gaze positions;
a processor, connected to the screen, the image capturing device, and the memory, and configured to obtain a head offset and an eye-gaze position of the user with respect to the screen from the facial image, determine whether to correct the facial image according to the head offset and the eye-gaze position, and correct the facial image based on the eye-gaze position and the preset codebook to generate a corrected facial image in response to the facial image determined to be corrected, wherein the head offset is calculated according to a configuration position of the image capturing device and a size of the screen.

14. The system according to claim 13 , wherein the memory further stores the configuration position of the image capturing device and the size of the screen, and the processor calculates a vertical distance and an orientation of the user with respect to the screen according to an area of a facial region in the facial image, the configuration position of the image capturing device, and the size of the screen so as to obtain the head offset.

15. The system according to claim 14 , wherein the memory stores a first correspondence relationship between distances and areas, wherein the first correspondence relationship records average facial areas with respect

Assignees

• Wistron Corp
• Winstron Corp

Inventors

• Lai Chin-Lun
• Chen Jun-Horng
• Yang Jie-Ci
• Kao Meng-Chao

Classifications

• H04N7/141 Primary

  between two video terminals, e.g. videophone (telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085) · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04N7/15

  Conference systems · CPC title

• G06V40/165 Primary

  using facial parts and geometric relationships · CPC title

• H04N5/2628 Primary

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title