Eye contact correction in real time using machine learning

Patent metadata

Field	Value
Publication number	US-10664949-B2
Application number	US-201615136621-A
Country	US
Kind code	B2
Filing date	Apr 22, 2016
Priority date	Apr 22, 2016
Publication date	May 26, 2020
Grant date	May 26, 2020

Abstract

Techniques related to eye contact correction to provide a virtual user gaze aligned with a camera while the user views a display are discussed. Such techniques may include determining and reducing histogram of oriented gradient features for an eye region of a source image to provide a feature set, applying a pretrained classifier to the feature set to determine a motion vector field for the eye region, and warping and inserting the eye region into the source image to generate an eye contact corrected image.

First claim

What is claimed is:

1. A machine implemented method for providing eye contact correction comprising:
determining histogram of oriented gradient features for an eye region of a source image;
reducing the histogram of oriented gradient features to provide a feature set corresponding to the eye region of the source image by applying principal component analysis to the histogram of oriented gradient features using pretrained scaling and weighting factors trained based on a plurality of pairs of training eye region images comprising first images having a gaze angle difference with respect to second images of the pairs of training eye region images, generation of a likelihood map for each pixel of each of the first images of the pairs of training eye region images, wherein each likelihood map comprises a sum of absolute differences for each of a plurality of candidate motion vectors corresponding to each pixel, determination of training stage histogram of oriented gradient features for the first images of the pairs of training eye region images, and application of principal component analysis to reduce the training stage histogram of oriented gradient features to a plurality of training stage feature sets corresponding to the first images of the pairs of training eye region images and to determine the pretrained scaling and weighting factors of the principal component analysis;
applying a pretrained random forest classifier to the entire feature set to determine a leaf of the pretrained random forest classifier that correlates the feature set to a motion vector field for the eye region of the source image; and
warping the eye region of the source image based on the motion vector field and integrating the warped eye region into a remaining portion of the source image to generate an eye contact corrected image.

2. The method of claim 1 , further comprising:
providing face detection and face landmark detection on the source image; and
cropping the source image based on the face detection and the face landmark detection to generate the eye region.

3. The method of claim 1 , further comprising:
encoding and transmitting the eye contact corrected image to a remote device for presentment to a user.

4. The method of claim 1 , wherein said reducing the histogram of oriented gradient features and said applying the pretrained random forest classifier to the feature set to determine the leaf of the pretrained random forest classifier that correlates the feature set to the motion vector field for the eye region of the source image are selectively provided based on a camera and a display having a first relative position therebetween, the method further comprising, when the camera and the display have a second relative position therebetween:
reducing the histogram of oriented gradient features to provide a second feature set corresponding to the eye region of the source image;
applying a second pretrained classifier to the second feature set to determine a second motion vector field for the eye region of the source image; and
warping the eye region of the source image based on the second motion vector field and integrating the warped eye region into the remaining portion of the source image to generate the eye contact corrected image.

5. The method of claim 1 , wherein the pretrained random forest classifier is generated based on a training stage pretrained classifier trained based on the likelihood maps and the training stage feature sets.

6. The method of claim 5 , wherein the pretrained random forest classifier is further generated based on compression of the training stage pretrained classifier by parameterized surface fitting to generate the pretrained random forest classifier.

7. A system for providing eye contact correction comprising:
a memory configured to store a source image; and
a processor coupled to the memory, the processor to:
determine histogram of oriented gradient features for an eye region of a source image;
reduce the histogram of oriented gradient features to provide a feature set corresponding to the eye region of the source image by the processor to apply principal component analysis to the histogram of oriented gradient features using pretrained scaling and weighting factors trained based on a plurality of pairs of training eye region images comprising first images of the pairs having a gaze angle difference with respect to second images of the pairs of training eye region images, generation of a likelihood map for each pixel of each of the first images of the pairs of training eye region images, wherein each likelihood map comprises a sum of absolute differences for each of a plurality of candidate motion vectors corresponding to each pixel, determination of training stage histogram of oriented gradient features for the first images of the pairs of training eye region images, and application of principal component analysis to reduce the training stage histogram of oriented gradient features to a plurality of training stage feature sets corresponding to the first images of the pairs of training eye region images and to determine the pretrained scaling and weighting factors of the principal component analysis;
apply a pretrained random forest classifier to the entire feature set to determine a leaf of the pretrained random forest classifier that correlates the feature set to a motion vector field for the eye region of the source image; and
warp the eye region of the source image based on the motion vector field and integrate the warped eye region into a remaining portion of the source image to generate an eye contact corrected image.

8. The system of claim 7 , wherein to reduce the histogram of oriented gradient features and to apply the pretrained random forest classifier to the feature set to determine the leaf of the pretrained random forest classifier that correlates the feature set to the motion vector field for the eye region of the source image are selectively provided based on a camera and a display having a first relative position therebetween, the processor further, when the camera and the display have a second relative position therebetween, to reduce the histogram of oriented gradient features to provide a second feature set corresponding to the eye region of the source image, to apply a second pretrained classifier to the second feature set to determine a second motion vector field for the eye region of the source image, and to warp the eye region of the source image based on the second motion vector field and integrate the warped eye region into the remaining portion of the source image to generate the eye contact corrected image.

9. The system of claim 7 , wherein the pretrained random forest classifier is generated based on a training stage pretrained classifier trained based on the likelihood maps and the training stage feature sets, and compression of the training stage pretrained classifier by parameterized surface fitting to generate the pretrained classifier.

10. At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a device, cause the device to provide eye contact correction by:
determining histogram of oriented gradient features for an eye region of a source image;
reducing the histogram of oriented gradient features to provide a feature set corresponding to the eye region of the source image by applying principal component analysis to the histogram of oriented gradient features using pretrained scaling and weighting factors trained based on a plurality of pairs of training eye region images comprising first images having a gaze angle difference with respect to second images of the pairs of training eye region

Assignees

• Intel Corp

Inventors

• Chalom Edmond
• Shimshi Or

Classifications

• H04N7/147

  Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q) · CPC title

• G06K9/6212

  Physics · mapped topic

• G06K9/00617

  Physics · mapped topic

• G06K9/6282

  Physics · mapped topic

• G06K9/66

  Physics · mapped topic