Image gaze correction method, apparatus, electronic device, computer-readable storage medium, and computer program product

Patent metadata

Field	Value
Publication number	US-12346494-B2
Application number	US-202217977682-A
Country	US
Kind code	B2
Filing date	Oct 31, 2022
Priority date	Jan 22, 2021
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

An image gaze correction method, apparatus, electronic device, computer-readable storage medium, and computer program product. The image gaze correction method includes: acquiring a to-be-corrected eye image from a to-be-corrected image, generating, based on the to-be-corrected eye image, an eye motion flow field and an eye contour mask, the eye motion flow field being used for adjusting a pixel position in the to-be-corrected eye image, and the eye contour mask being used for indicating a probability that the pixel position in the to-be-corrected eye image belongs to an eye region, performing, based on the eye motion flow field and the eye contour mask, gaze correction processing on the to-be-corrected eye image to obtain a corrected eye image, and generating a gaze corrected image based on the corrected eye image.

First claim

What is claimed is:

1. A method comprising:
acquiring, by a computing device, a to-be-corrected eye image from a to-be-corrected image;
generating, by the computing device and based on the to-be-corrected eye image, an eye motion flow field comprising a first plurality of pixels and an eye contour mask comprising a second plurality of pixels, wherein each of the first plurality of pixels comprises a horizontal displacement and a vertical displacement and each of the second plurality of pixels indicates a probability that a corresponding pixel in the to-be-corrected eye image belongs to an eye region;
performing, by the computing device and based on the eye motion flow field and the eye contour mask, gaze correction processing on the to-be-corrected eye image to obtain a corrected eye image, the gaze correction processing comprising:
performing, using the eye motion flow field, transformation processing on the to-be-corrected eye image to obtain an initially corrected eye image;
performing, using the eye contour mask, adjustment processing on the initially corrected eye image to obtain a further corrected eye image, wherein the adjustment processing comprises fusing pixel values of corresponding positions in the eye contour mask and the initially corrected eye image to obtain a first intermediate image; and

generating, by the computing device, a gaze corrected image based on the corrected eye image.

2. The method according to claim 1 , wherein the gaze correction processing further comprises:
determining the further corrected eye image as the corrected eye image.

3. The method according to claim 2 , wherein the adjustment processing further comprises:
fusing pixel values of corresponding positions in a mapped image corresponding to the eye contour mask and the to-be-corrected eye image to obtain a second intermediate image; and
integrating pixel values of corresponding positions in the first intermediate image and the second intermediate image to obtain the further corrected eye image.

4. The method according to claim 1 , further comprising:
performing down-sampling processing on the to-be-corrected eye image to generate a down-sampled image; and
performing up-sampling processing on the down-sampled image to obtain the eye motion flow field and the eye contour mask, wherein during the up-sampling processing, a feature map generated by the up-sampling processing and a feature map of a same size generated during the down-sampling processing are cascaded and then used as an input image of a next up-sampling operation.

5. The method according to claim 4 , wherein the down-sampling processing and the up-sampling processing are implemented by a gaze correction model, the eye motion flow field is data of a first channel and a second channel output data of the gaze correction model, and the eye contour mask is data of a third channel in the output data.

6. The method according to claim 1 , wherein the acquiring comprises:
recognizing contour key points of eyes from the to-be-corrected image;
determining a minimum circumscribed rectangle based on the contour key points;
specifying a multiple for extension of the minimum circumscribed rectangle to obtain an image capture frame; and
capturing, based on the image capture frame, the to-be-corrected eye image from the to-be-corrected image.

7. The method according to claim 1 , wherein the generating the gaze corrected image comprises:
integrating the corrected eye image into an image capture frame position of the to-be-corrected image to obtain an integrated image, wherein the image capture frame position is a position of the to-be-corrected eye image in the to-be-corrected image; and
performing image harmonization processing at the image capture frame position in the integrated image to obtain the gaze corrected image, wherein the image harmonization processing is used for eliminating boundary traces at the image capture frame position.

8. The method according to claim 7 , wherein the performing image harmonization processing at the image capture frame position in the integrated image comprises:
generating an initialized mask image of a same size as the to-be-corrected image, wherein a pixel value of the initialized mask image at the image capture frame position is a first specified value, pixel values of remaining positions are a second specified value different from the first specified value, and the remaining positions are positions in the initialized mask image except the image capture frame position;
performing noise processing on the initialized mask image to obtain a processed mask image;
fusing pixel values of corresponding positions in the processed mask image and the integrated image to obtain a first generated image;
fusing pixel values of corresponding positions in a mapped image corresponding to the processed mask image and the to-be-corrected image to obtain a second generated image; and
integrating pixel values of corresponding positions in the first generated image and the second generated image to obtain the gaze corrected image.

9. An apparatus comprising:
one or more processors; and
memory storing computer-readable instructions that when executed by the one or more processors, cause the apparatus to:
acquire a to-be-corrected eye image from a to-be-corrected image;
generate, based on the to-be-corrected eye image, an eye motion flow field comprising a first plurality of pixels and an eye contour mask comprising a second plurality of pixels, wherein each of the first plurality of pixels comprises a horizontal displacement and a vertical displacement, and each of the second plurality of pixels indicates a probability that a corresponding pixel in the to-be-corrected eye image belongs to an eye region;
perform, based on the eye motion flow field and the eye contour mask, gaze correction processing on the to-be-corrected eye image to obtain a corrected eye image, the gaze correction processing comprising:
performing, using the eye motion flow field, transformation processing on the to-be-corrected eye image to obtain an initially corrected eye image;
performing, using the eye contour mask, adjustment processing on the initially corrected eye image to obtain a further corrected eye image, wherein the adjustment processing comprises fusing pixel values of corresponding positions in the eye contour mask and the initially corrected eye image to obtain a first intermediate image; and

generate a gaze corrected image based on the corrected eye image.

10. The apparatus according to claim 9 , wherein the gaze correction processing further comprises:
determining the further corrected eye image as the corrected eye image.

11. The apparatus according to claim 10 , wherein the adjustment processing further comprises:
fusing pixel values of corresponding positions in a mapped image corresponding to the eye contour mask and the to-be-corrected eye image to obtain a second intermediate image; and
integrating pixel values of corresponding positions in the first intermediate image and the second intermediate image to obtain the further corrected eye image.

12. The apparatus according to claim 9 , the memory storing computer-readable instructions that when executed by the one or more processors, further cause the apparatus to:
perform down-sampling processing on the to-be-corrected eye image to generate a down-sampled image; and
perform up-sampling processing on the down-sampled image to obtain the eye motion flow field and the eye contour mask, wherein during the up-sampling processing, a feature map generated by the up-sampling processing and a feature map of a same size generated during the dow

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Peng Jinlong
• He Keke
• Jiang Zhengkai
• Yi Yang
• Yu Xiaoming
• Tu Juanhui
• Zhou Yi
• Liu Chenghao
• Wang Yabiao
• Tai Ying
• Wang Chengjie
• Li Jilin
• Huang Feiyue

Classifications

• G06V10/46

  Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) · CPC title

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V10/771

  Feature selection, e.g. selecting representative features from a multi-dimensional feature space · CPC title