Gaze correction method and apparatus for face image, device, computer-readable storage medium, and computer program product face image

Patent metadata

Field	Value
Publication number	US-12450688-B2
Application number	US-202217977576-A
Country	US
Kind code	B2
Filing date	Oct 31, 2022
Priority date	Jan 22, 2021
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

In the field of artificial intelligence technologies, a gaze correction method and apparatus for a face image, a device, a computer-readable storage medium, and a computer program product are provided. The method includes: acquiring an eye image from a face image; determining an eye movement flow field based on the eye image and a target gaze direction, the target gaze direction being a gaze direction to which an eye gaze in the eye image is to be corrected; adjusting a pixel position in the eye image based on the eye movement flow field, to obtain a corrected eye image; and generating a face image with a corrected gaze based on the corrected eye image.

First claim

What is claimed is:

1. A gaze correction method, performed by a first computer device, the method comprising:
acquiring an eye image from a face image;
determining an eye movement flow field by inputting the eye image and a target gaze direction to a gaze correction model, the target gaze direction being a gaze direction to which an eye gaze in the eye image is to be corrected;
perform gaze correction on each pixel of the eye image based on a horizontal displacement and a vertical displacement of each pixel acquired from the eye movement flow field, to obtain a corrected eye image;
determining an eye contour mask by performing channel dimension-based combination on the eye image and the target gaze direction, and adjusting a pixel position in the corrected eye image by applying the eye contour mask, which is obtained based on the eye movement flow field, to obtain an adjusted eye image, the eye contour mask indicating a probability that the pixel position in the eye image is in an eye region; and
generating a face image with a corrected gaze based on the adjusted eye image.

2. The method according to claim 1 , wherein the determining the eye movement flow field comprises:
performing the channel dimension-based combination on the eye image and the target gaze direction to obtain combined data;
performing feature extraction on the combined data by using a the gaze correction model, to obtain output data of the gaze correction model; and
extracting the eye movement flow field from the output data.

3. The method according to claim 1 , wherein the determining the eye contour mask comprises:
performing the channel dimension-based combination on the eye image and the target gaze direction to obtain combined data;
performing feature extraction on the combined data by using a the gaze correction model, to obtain output data of the gaze correction model; and
extracting the eye contour mask from the output data.

4. The method according to claim 1 , wherein the adjusting the pixel position in the corrected eye image comprises:
multiplying a pixel value of the eye contour mask by a pixel value in the corrected eye image corresponding to a same position, to obtain a first intermediate image;
multiplying a pixel value of a mapped image corresponding to the eye contour mask by a pixel value in the eye image corresponding to a same position, to obtain a second intermediate image; and
summing pixel values in the first intermediate image and the second intermediate image corresponding to the same position, to obtain the adjusted eye image.

5. The method according to claim 1 , further comprising training, by a second computer device, the gaze correction model wherein the gaze correction model is trained by:
training a movement-flow-field-based first teacher gaze correction model by using an eye image sample, to obtain a trained first teacher gaze correction model, the trained first teacher gaze correction model being configured to output an eye movement flow field of the eye image sample, wherein a pixel position in the eye image sample is to be adjusted based on the eye movement flow field;
training an image-based second teacher gaze correction model by using the eye image sample, to obtain a trained second teacher gaze correction model, the trained second teacher gaze correction model being configured to output a corrected eye image sample of the eye image sample; and
performing knowledge distillation training on a student gaze correction model by using the trained first teacher gaze correction model and the trained second teacher gaze correction model, to obtain a trained student gaze correction model.

6. The method according to claim 5 , wherein the training the movement- flow-field-based first teacher gaze correction model comprises:
acquiring a training sample of the first teacher gaze correction model, the training sample comprising the eye image sample and a target corrected eye image;
performing eye feature extraction on the eye image sample by using the first teacher gaze correction model, to obtain the eye movement flow field and an eye contour mask of the eye image sample, the eye contour mask indicating a probability that the pixel position in the eye image sample is in an eye region;
determining a corrected eye image sample based on the eye image sample, a corresponding eye movement flow field, and a corresponding eye contour mask; and
constructing a loss function of the first teacher gaze correction model based on the corrected eye image sample and the target corrected eye image, and adjusting a parameter of the first teacher gaze correction model based on the loss function of the first teacher gaze correction model.

7. The method according to claim 5 , wherein the training the image-based second teacher gaze correction model comprises:
acquiring a training sample of the second teacher gaze correction model, the training sample comprising the eye image sample and a target corrected eye image;
performing gaze correction on the eye image sample by using the second teacher gaze correction model, to obtain a corrected eye image sample and an eye contour mask, the eye contour mask indicating a probability that the pixel position in the eye image sample is in an eye region;
adjusting the corrected eye image sample based on the eye contour mask, to obtain an adjusted eye image sample; and
constructing a loss function of the second teacher gaze correction model based on the adjusted eye image sample and the target corrected eye image, and adjusting a parameter of the second teacher gaze correction model based on the loss function of the second teacher gaze correction model.

8. The method according to claim 5 , wherein the performing the knowledge distillation training comprises:
acquiring a training sample of the student gaze correction model, the training sample comprising the eye image sample and a target corrected eye image;
outputting a teacher eye movement flow field and a first teacher eye contour mask of the eye image sample by using the trained first teacher gaze correction model, and generating a first output image based on the eye image sample, a corresponding teacher eye movement flow field, and a corresponding first teacher eye contour mask;
outputting a corrected image and a second teacher eye contour mask of the eye image sample by using the trained second teacher gaze correction model, and generating a second output image based on the corrected image and the second teacher eye contour mask;
outputting a student eye movement flow field and a student eye contour mask of the eye image sample by using the student gaze correction model, and generating a third output image based on the eye image sample, a corresponding student eye movement flow field, and a corresponding student eye contour mask;
constructing a loss function of the student gaze correction model based on a difference between the first output image and the third output image, a difference between the second output image and the third output image, and a difference between the third output image and the target corrected eye image; and
adjusting a parameter of the student gaze correction model based on the loss function of the student gaze correction model, to obtain a the trained student gaze correction model.

9. The method according to claim 8 , wherein the generating the third output image comprises:
transforming the eye image sample based on the student eye movement flow field, to obtain a transformed image; and
adjusting the transformed image by using the student eye contour mask, to obtain the third output image.

10. The method according to claim 8 , wherein the constructing the loss function comprises:
determ

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Jiang Zhengkai
• Peng Jinlong
• He Keke
• Yu Xiaoming
• Yi Yang
• Tu Juanhui
• Zhou Yi
• Liu Chenghao
• Wang Yabiao
• Tai Ying
• Wang Chengjie
• Li Jilin
• Huang Feiyue

Classifications

• G06T2207/30201

  Face · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title