Image gaze correction method, apparatus, electronic device, computer-readable storage medium, and computer program product

Patent metadata

Field	Value
Publication number	US-12197640-B2
Application number	US-202217977646-A
Country	US
Kind code	B2
Filing date	Oct 31, 2022
Priority date	Jan 22, 2021
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

An image gaze correction method, apparatus, electronic device, computer-readable storage medium, and computer program product related to the field of artificial intelligence technologies are provided. The image gaze correction method includes: acquiring an eye image from an image; performing feature extraction processing on the eye image to obtain feature information of the eye image; performing, based on the feature information and a target gaze direction, gaze correction processing on the eye image to obtain an initially corrected eye image and an eye contour mask; performing, by using the eye contour mask, adjustment processing on the initially corrected eye image to obtain a corrected eye image; and generating a gaze corrected image based on the corrected eye image.

First claim

What is claimed is:

1. An image gaze correction method, the method being performed by an electronic device, and the method comprising:
acquiring an eye image from an image;
performing feature extraction processing on the eye image to obtain feature information of the eye image;
performing, based on the feature information and a target gaze direction, gaze correction processing on the eye image to obtain an initially corrected eye image and an eye contour mask, the target gaze direction being a gaze direction to which an eye gaze in the eye image is to be corrected, and a pixel value of each position in the eye contour mask having a value in a range from 0 to 1 indicating a probability that a corresponding pixel position in the eye image belongs to an eye region;
performing, by using the eye contour mask, adjustment processing on the initially corrected eye image to obtain a corrected eye image; and
generating a gaze corrected image based on the corrected eye image,
wherein the performing the adjustment processing on the initially corrected eye image comprises:
fusing pixel values of corresponding positions in the eye contour mask and the initially corrected eye image to obtain a first intermediate image;
fusing pixel values of corresponding positions in a mapped image corresponding to the eye contour mask and the eye image to obtain a second intermediate image, wherein a pixel value of any position in the mapped image is a value obtained by subtracting a pixel value at a corresponding position in the eye contour mask from 1; and
integrating pixel values of corresponding positions in the first intermediate image and the second intermediate image to obtain the corrected eye image.

2. The method according to claim 1 , wherein the performing the feature extraction processing comprises:
performing the feature extraction processing on the eye image to obtain an eye expression feature, an eye expression irrelevant feature, and an environment-related feature; and
determining the eye expression feature, the eye expression irrelevant feature, and the environment-related feature as the feature information.

3. The method according to claim 1 , wherein the performing the gaze correction processing comprises:
combining the feature information and the target gaze direction in a channel dimension to obtain combined data; and
performing, based on a feature dimension of the eye image, feature reconstruction on the combined data to obtain the initially corrected eye image and the eye contour mask.

4. The method according to claim 1 , wherein the generating the gaze corrected image comprises:
integrating the corrected eye image into an image capture frame position of the image to obtain an integrated image, wherein the image capture frame position is a position of the eye image in the image; and
performing image harmonization processing to eliminate boundary traces at the image capture frame position in the integrated image, and obtaining the gaze corrected image based on a result of the image harmonization processing.

5. The method according to claim 1 , wherein the feature extraction processing and the gaze correction processing are performed by using a gaze correction model, wherein the gaze correction model comprises an encoding network and a decoding network, the encoding network is configured to perform the feature extraction processing, and the decoding network is configured to perform the gaze correction processing.

6. The method according to claim 5 , wherein the gaze correction model is trained by:
acquiring a training sample for training the gaze correction model, the training sample comprising an eye image sample and a target corrected eye image;
performing, by the gaze correction model, gaze correction processing on the eye image sample, the target corrected eye image, and a target gaze direction sample to obtain an initially corrected eye image sample and an eye contour mask sample, the target gaze direction sample being a gaze direction to which an eye gaze in the eye image sample is to be corrected, and a pixel value of each position in the eye contour mask having a value in a range from 0 to 1 indicating a probability that a corresponding pixel position in the eye image sample belongs to an eye region;
performing, by using the eye contour mask sample, adjustment processing on the initially corrected eye image sample to obtain a corrected eye image sample; and
determining, based on the corrected eye image sample and the target corrected eye image, a loss of the gaze correction model, and adjusting a parameter of the gaze correction model based on the loss to obtain the gaze correction model.

7. The method according to claim 6 , wherein the gaze correction model comprises a first encoding network, a second encoding network, and a decoding network; and
the performing, by the gaze correction model, the gaze correction processing comprises:
performing, by the first encoding network, feature extraction processing on the eye image sample to obtain an eye expression feature sample and an eye expression irrelevant feature sample;
performing, by the second encoding network, feature extraction processing on the target corrected eye image to obtain a target environmental feature; and
performing, by the decoding network, the gaze correction processing on the eye expression feature sample, the eye expression irrelevant feature sample, the target environmental feature, and the target gaze direction sample to obtain the initially corrected eye image sample and the eye contour mask sample.

8. The method according to claim 6 , wherein the determining the loss of the gaze correction model comprises:
determining a reconstruction loss based on a pixel difference between the corrected eye image sample and the target corrected eye image;
determining a feature loss based on an image feature difference between the corrected eye image sample and the target corrected eye image;
determining a generative adversarial loss between the corrected eye image sample and the target corrected eye image; and
determining the loss of the gaze correction model based on the reconstruction loss, the feature loss, and the generative adversarial loss.

9. The method according to claim 8 , wherein the determining the feature loss based on the image feature difference between the corrected eye image sample and the target corrected eye image comprises:
outputting, by a feature loss calculation model, a learned perceptual image patch similarity (LPIPS) loss between the corrected eye image sample and the target corrected eye image, wherein the feature loss comprises the LPIPS loss.

10. The method according to claim 8 , wherein the determining the generative adversarial loss between the corrected eye image sample and the target corrected eye image comprises:
determining a generative network loss and a discriminant network loss based on a discrimination result of a multi-scale discriminator on the corrected eye image sample and the target corrected eye image; and
determining the generative network loss and the discriminant network loss as the generative adversarial loss.

11. The method according to claim 8 , wherein the method further comprises:
performing gaze estimation on the corrected eye image sample to obtain a gaze direction of the corrected eye image sample; and
determining a gaze estimation loss based on the gaze direction of the corrected eye image sample and the target gaze direction sample; and
the determining the loss of the gaze correction model based on the reconstruction loss, the feature loss, and the generative adversarial loss comprises:
determining the loss of the gaze correction model base

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• He Keke
• Jiang Zhengkai
• Peng Jinlong
• Yi Yang
• Yu Xiaoming
• Tu Juanhui
• Zhou Yi
• Wang Yabiao
• Tai Ying
• Wang Chengjie
• Li Jilin
• Huang Feiyue

Classifications

• G06V10/46

  Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V40/165 Primary

  using facial parts and geometric relationships · CPC title