Feature detection and masking in images based on color distributions

Patent metadata

Field	Value
Publication number	US-9864901-B2
Application number	US-201514855297-A
Country	US
Kind code	B2
Filing date	Sep 15, 2015
Priority date	Sep 15, 2015
Publication date	Jan 9, 2018
Grant date	Jan 9, 2018

Abstract

Implementations relate to image feature detection and masking in images based on color distributions. In some implementations, a computer-implemented method to determine a mask for an image includes determining a spatial function for a detected feature depicted in the image, the function indicating pixels of the image relative to an estimated feature boundary. A respective color likelihood distribution is determined for each of multiple regions of the image based on one or more distributions of color values in the regions, including a feature region and a non-feature region. A confidence mask is determined based on the spatial function and one or more of the color likelihood distributions. The confidence mask indicates, for each of multiple pixels of the image, an associated confidence that the pixel is a feature pixel. A modification is applied to pixels in the image using the confidence mask.

First claim

What is claimed is:

1. A computer-implemented method to determine a mask for an image, the method comprising:
determining a spatial function for a detected feature depicted in the image, wherein the detected feature is a detected skin feature, wherein the spatial function indicates pixels of the image relative to a feature boundary estimated from the detected feature, wherein determining the spatial function includes determining a prior probability mask for the detected feature, wherein the prior probability mask is based on the feature boundary and indicates, for each of a first plurality of pixels of the image, an associated first probability that the pixel is a feature pixel;
determining a respective color likelihood distribution for each region of a plurality of regions of the image based on one or more distributions of color values in each of the regions, wherein the regions include a feature region and a non-feature region;
determining a confidence mask based on the spatial function and one or more of the color likelihood distributions, wherein the confidence mask indicates, for each of a second plurality of pixels of the image, an associated confidence that the pixel is a feature pixel, wherein the confidence mask is based on a posterior probability mask, wherein the posterior probability mask is based on the prior probability mask and the one or more color likelihood distributions, wherein the posterior probability mask indicates, for each of the second plurality of pixels of the image, an associated second probability that the pixel is a feature pixel,
wherein determining the confidence mask is based on a posterior feature mask and a posterior non-feature mask, and includes:
determining the posterior feature mask based on the color likelihood distribution for the feature region, wherein the posterior feature mask indicates, for each of a third plurality of pixels of the image, an associated third probability that the pixel is a feature pixel; and
determining the posterior non-feature mask based on at least one of the one or more color likelihood distributions for the non-feature region, wherein the posterior non-feature mask indicates, for each of a fourth plurality of pixels of the image, an associated fourth probability that the pixel is not a feature pixel; and

applying a modification to one or more pixels of the image using the confidence mask.

2. The method of claim 1 , further comprising blurring the spatial function, wherein determining the confidence mask is based on the blurred spatial function.

3. The method of claim 1 , wherein the detected skin feature is facial skin included in a detected face, wherein the detected face includes one or more detected facial landmarks, wherein the feature boundary is a face boundary estimated based on the one or more detected facial landmarks.

4. The method of claim 3 , further comprising determining the face boundary using a machine learning technique that determines the face boundary based on positions of the one or more detected facial landmarks, wherein the machine learning technique is trained with a plurality of training images having face boundaries indicated.

5. The method of claim 1 , wherein determining a respective color likelihood distribution for each region includes:
determining a region segmentation of the image based on the feature boundary, wherein the region segmentation indicates a plurality of regions including the feature region and the non-feature region of the image;
determining the one or more distributions of color values for pixels of the image in each of the plurality of regions; and
determining the respective color likelihood distribution for each of the plurality of regions based on one or more of the distributions of color values for the regions.

6. The method of claim 1 , wherein the distributions of color values include histograms, and further comprising normalizing the histograms for the regions.

7. The method of claim 1 further comprising blurring one or more of the distributions of color values.

8. The method of claim 7 wherein blurring the distributions of color values includes blurring values of the one or more distributions along an axis for which color value for different color channels are approximately equal.

9. The method of claim 1 wherein determining the posterior probability mask includes multiplying the prior probability mask with at least one of the color likelihood distributions.

10. The method of claim 1 wherein determining the confidence mask includes determining a ratio of the posterior feature mask to the posterior non-feature mask.

11. The method of claim 1 wherein the detected feature is facial skin included in a detected face, and wherein the plurality of regions include one or more facial landmark regions different than the feature region, wherein the one or more facial landmark regions include at least one of: a mouth region, an eye region, and an eyebrow region, and
wherein determining the respective color likelihood distribution for each region includes determining a respective color likelihood distribution for each of the one or more facial landmark regions.

12. The method of claim 11 , wherein the posterior non-feature mask is based on multiplying the prior probability mask with a combination of the color likelihood distributions for the non-feature region and for the one or more facial landmark regions.

13. A system to determine a mask for an image, the system comprising:
a storage device; and
at least one processor operative to access the storage device and configured to:
determine a respective color likelihood distribution for each region of a plurality of regions of the image based on one or more distributions of color values in each of the regions, wherein the regions include a feature region estimated to depict at least a portion of an image feature and a non-feature region estimated to not depict the image feature;
determine a probability feature mask at least based on the color likelihood distribution for the feature region, wherein the probability feature mask indicates, for each of a first plurality of pixels of the image, an associated probability that the pixel is a feature pixel;
determine a probability non-feature mask at least based on the color likelihood distribution for the non-feature region, wherein the probability non-feature mask indicates, for each of a second plurality of pixels of the image, an associated probability that the pixel is not a feature pixel;
determine a resulting confidence mask based on the probability feature mask and the probability non-feature mask; and
apply a modification to one or more pixels of the image using the resulting confidence mask.

14. The system of claim 13 wherein the one or more distributions of color values include histograms, and wherein the at least one processor is further configured to blur each the histograms.

15. The system of claim 13 wherein the image feature is facial skin included in a detected face, and wherein the plurality of regions include one or more facial landmark regions, wherein the one or more facial landmark regions include at least one of: a mouth region, an eye region, and an eyebrow region, and
wherein the at least one processor is configured to determine a respective color likelihood distribution for each of the one or more facial landmark regions, and
wherein the at least one processor is configured to determine the probability non-feature mask for the facial skin at least based on the color likelihood distribution for the non-feature region and the color likelihood distributions for the one or more

Assignees

• Google Inc
• Google Llc

Inventors

• Chang Jason

Classifications

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06V40/171 Primary

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06K9/627

  Physics · mapped topic