Detecting and modifying facial features of persons in images

Patent metadata

Field	Value
Publication number	US-9256950-B1
Application number	US-201414200000-A
Country	US
Kind code	B1
Filing date	Mar 6, 2014
Priority date	Mar 6, 2014
Publication date	Feb 9, 2016
Grant date	Feb 9, 2016

Abstract

Implementations relate to detecting and modifying facial features of persons in images. In some implementations, a method includes receiving one or more general color models of color distribution for a facial feature of persons depicted in training images. The method obtains an input image, and determines a feature mask associated with the facial feature for one or more faces in the input image. Determining the mask includes estimating one or more local color models for each of the faces in the input image based on the general color models, and iteratively refining the estimated local color models based on the general color models. The refined local color models are used in the determination of the feature mask. The method applies a modification to the facial feature of faces in the input image using the feature mask.

First claim

What is claimed is:

1. A method comprising:
receiving one or more general color models of color distribution for a facial feature of persons depicted in a plurality of training images;
obtaining an input image;
determining a probability location mask in a feature region for each of one or more faces detected in the input image, the probability location mask indicating probabilities based on locations in the feature region and providing a higher probability at the center of the feature region than at the edges;
determining a feature mask associated with the facial feature for the one or more faces detected in the input image, wherein determining the feature mask uses the one or more general color models and the probability location mask and includes:
estimating one or more local color models for each of the one or more faces in the input image based on the one or more general color models; and
refining the one or more estimated local color models for each of the one or more faces in the image based on the one or more general color models,

wherein the refined one or more local color models are used in the determination of the feature mask; and
applying a modification to the facial feature of at least one of the one or more faces in the input image using the feature mask.

2. The method of claim 1 wherein the facial feature is teeth.

3. The method of claim 1 wherein the of one or more general color models are determined and stored before the obtaining of the input image, and wherein the one or more general color models are Gaussian models including a first model for the colors of feature pixels in the facial feature and a second model for the colors of non-feature pixels that are not in the facial feature in the plurality of training images.

4. The method of claim 1 wherein the facial feature is teeth, and wherein the probability location mask is centered on a center of a detected mouth region.

5. The method of claim 4 wherein the probability location mask is a two-dimensional Gaussian gradient distribution having a probability of 1 in the center of the probability location mask and probabilities gradually falling to zero at the edges of the mouth region.

6. The method of claim 1 wherein estimating one or more local color models includes evaluating each pixel in a feature region of each of the one or more faces with the one or more general color models to determine a local feature color model for pixels depicting the facial feature and a local non-feature color model for pixels not depicting the facial feature.

7. The method of claim 6 wherein a pixel color of the input image is added to the local feature color model if the probability of the pixel being a feature pixel is over a predetermined threshold, and the pixel color is added to the local non-feature color model if the probability is under the predetermined threshold.

8. The method of claim 1 wherein the refining the one or more estimated local color models includes comparing and transforming data points of the one or more estimated local color models to the one or more general color models.

9. The method of claim 1 wherein the refining includes repeating for a number of times:
transforming the one or more local color models by aligning the centers and major bases of the one or more local color models and one or more of the corresponding general color models; and
evaluating each pixel in a feature region of a face using pixel values from the one or more transformed local color models fed to the one or more Response After Final corresponding general color models to determine a new local color model for feature pixels and a new local color model for non-feature pixels.

10. The method of claim 1 wherein one or more resulting color models result from the refining, and wherein determining the feature mask associated with each of the one or more faces includes using at least one of the one or more resulting color models to determine mask values in an individual feature mask for each of the one or more faces depicted in the input image.

11. The method of claim 10 wherein determining the feature mask includes merging the individual feature mask determined for each of the one or more faces into the feature mask.

12. A method comprising:
receiving one or more general color models of color distribution for a facial feature of persons depicted in a plurality of training images, wherein the facial feature is teeth;
obtaining an input image;
determining a probability location mask in a feature region, for each of one or more faces detected in the input image, the probability location mask indicating probabilities based on locations in the feature region;
determining a feature mask associated with the facial feature for the one or more faces detected in the input image, wherein determining the feature mask uses the one or more general color models and the probability location mask and includes:
estimating one or more local color models for each of the one or more faces in the input image based on the one or more general color models; and
refining the one or more estimated local color models for each of the one or more faces in the image based on the one or more general color models,
wherein the refined one or more local color models are used in the determination of the feature mask; and

applying a modification to the facial feature of at least one of the one or more faces in the input image using the feature mask, wherein the applying a modification includes a color change that includes at least one of:
reducing a saturation value of each teeth pixel; and
increasing a brightness value of each teeth pixel.

13. A system comprising:
a storage device; and
at least one processor accessing the storage device and operative to perform operations comprising:
receiving one or more general color models of color distribution for a facial feature of persons depicted in a plurality of training images;
obtaining an input image;
determining a feature mask associated with the facial feature of one or more faces detected in the input image, including:
estimating one or more local color models for each of the one or more faces in the input image based on the one or more general color models; and
refining the one or more estimated local color models for each of the one or more faces in the image, including transforming data points of the one or more estimated local color models to the one or more general color models, wherein the refined one or more local color models are used in the determination of the feature mask; and

applying a modification to the facial feature of at least one of the one or more faces in the input image using the feature mask,
wherein the at least one processor refines the estimated one or more local color models for each of the one or more faces including repeating for a number of times:
transforming the data points of the one or more local color models by aligning the centers and major bases of the one or more local color models and one or more corresponding general color models; and
evaluating each pixel in a feature region of a face using pixel values from the one or more transformed local color models fed to the one or more corresponding general color models to determine a new local color model for feature pixels and a new local color model for non-feature pixels.

14. The system of claim 13 wherein the at least one processor is further operative to perform:
determining a probability location mask in a feature region for each of the one or more faces detected in the input image, wherein the probability location mask provides a higher probabil

Assignees

• Google Inc

Inventors

• Xu Hao
• Diverdi Stephen Joseph
• Babacan Sevket Derin
• Krishnaswamy Aravind
• Liu Tinglin

Classifications

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06T7/0081 Primary

  Physics · mapped topic

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title