Face image processing methods and apparatuses, and electronic devices

Patent metadata

Field	Value
Publication number	US-11227147-B2
Application number	US-201916456738-A
Country	US
Kind code	B2
Filing date	Jun 28, 2019
Priority date	Aug 9, 2017
Publication date	Jan 18, 2022
Grant date	Jan 18, 2022

Abstract

A face image processing method includes: segmenting a face in an image to be processed to obtain at least one organ image block; respectively inputting the at least one organ image block into at least one first neural network, where at least two different types of organs correspond to different first neural networks; and extracting key point information of an organ from the respective input organ image block by the at least one first neural network to respectively obtain key point information of at least one corresponding organ of the face.

First claim

The invention claimed is:

1. A face image processing method, comprising:
segmenting a face in an image to be processed to obtain at least one organ image block;
respectively inputting the at least one organ image block into at least one first neural network, wherein at least two different types of organs correspond to different first neural networks; and
extracting key point information of an organ from the respective input organ image block by the at least one first neural network to respectively obtain key point information of at least one corresponding organ of the face,
wherein before the respectively inputting the at least one organ image block into at least one first neural network, the method further comprises:
training the first neural network based on a sample data set, wherein the sample data set comprises key point marking data of an organ image of the face,
wherein the key point marking data of the organ image of the face is obtained by:
determining a curve control point of an organ of the face;
forming a first curve according to the curve control point; and
inserting a plurality of points in the first curve by means of interpolation, wherein information of the inserted points is the key point marking data.

2. The method according to claim 1 , wherein the segmenting a face in an image to be processed to obtain at least one organ image block comprises:
obtaining initial face key point information of the image to be processed; and
segmenting the face in the image to be processed according to the initial face key point information to obtain the at least one organ image block, wherein the at least one organ image block comprises at least one of: at least one eye image block, or at least one mouth image block.

3. The method according to claim 1 , wherein the method further comprises:
obtaining initial face key point information of the image to be processed; and
integrating the initial face key point information and the key point information of the at least one corresponding organ to obtain face key point information of the image to be processed.

4. The method according to claim 3 , wherein the obtaining initial face key point information of the image to be processed comprises:
inputting the image to be processed into a second neural network; and
extracting, by the second neural network, face key point information of the image to be processed to obtain the initial face key point information.

5. The method according to claim 3 , wherein the integrating the initial face key point information and the key point information of the at least one corresponding organ to obtain face key point information of the image to be processed comprises:
replacing at least part of key point information of one organ of the initial face key point information with the key point information of the at least one corresponding organ to obtain the face key point information of the image to be processed, said one organ being the same as the at least one corresponding organ.

6. The method according to claim 3 , wherein the integrating the initial face key point information and the key point information of the at least one corresponding organ to obtain face key point information of the image to be processed comprises:
respectively converting a position and a serial number of the key point information of the at least one corresponding organ in the input respective organ image block into a position and a serial number of the key point information of the at least one corresponding organ in the image to be processed.

7. The method according to claim 3 , wherein
a total number of key points comprised in the face key point information is greater than a total number of key points comprised in the initial face key point information; and/or,
a number of organ key points of one organ image block extracted by the first neural network and comprised in the face key point information is greater than a number of organ key points corresponding to the organ image block comprised in the initial face key point information.

8. The method according to claim 7 , wherein an error degree of an organ curve fitted by at least two organ key points of the organ image block extracted by the first neural network is 1/10 to ⅕ of the error degree of an organ curve fitted by at least two organ key points corresponding to the organ image block comprised in the initial face key point information, wherein the error degree is an error degree of a fitted curve with respect to an actual organ curve of the face.

9. The method according to claim 2 , wherein the key point information of the at least one corresponding organ comprises at least one of: eyelid line information, or lip line information,
wherein the eyelid line information comprises: trajectory information or a fitted line represented by 10-15 key points at a monocular upper eyelid or lower eyelid;
the lip line information comprises: trajectory information or a fitted line represented by 16-21 key points at an upper contour of a single lip and 16-21 key points at a lower contour of the single lip.

10. The method according to claim 3 , wherein a number of the key points comprised in the initial face key point information is less than or equal to 106, and a number of the key points comprised in the face key point information is greater than 106.

11. The method according to claim 3 , wherein the number of the key points comprised in the face key point information is 186, 240, 220, or 274,
wherein the face key point information comprises:
4-24 key points for locating the eye position, and 44-48 key points comprised in the eyelid line information;
0-20 key points for locating the mouth position, and 60-64 key points comprised in the lip line;
26-58 key points comprised in an eyebrow area;
15-27 key points comprised in a nose area; and
33-37 key points of the face contour.

12. The method according to claim 1 , wherein the error degree of a second curve fitted by the inserted points with respect to the organ curve of the face is 1/10 to ⅕ of the error degree of the first curve with respect to the organ curve of the face, wherein the error degree is an error degree of a fitted curve with respect to an actual organ curve of the face.

13. The method according to claim 1 , further comprising:
performing at least one of the following processing according to the obtained key point information of the particular organ of the face: image rendering of the face, face changing, beautifying processing, makeup processing, face recognition, face state detection, facial expression detection, or attribute detection.

14. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by a processor, cause the processor to perform operations of the face image processing method according to claim 1 .

15. A face image processing method, comprising:
obtaining an image to be processed comprising at least a partial area of a face; and
extracting, by a neural network, eyelid line information or lip line information from the image to be processed, wherein the eyelid line information comprises: trajectory information or a fitted line represented by 10-15 key points at a monocular upper eyelid or lower eyelid, and wherein the lip line information comprises: trajectory information or a fitted line represented by 16-21 key points at an upper contour of a single lip and 16-21 key points at a lower contour of the single lip,
wherein before the extracting, by a neural network, eyelid line information from the image to be processed, the method further comprises:
training the neural

Assignees

• Beijing Sensetime Tech Development Co Ltd

Inventors

• Qian Chen
• Wang Quan
• Wang Fei

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/84

  using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title