Modifying digital content with digital effects using facial skin mask

Patent metadata

Field	Value
Publication number	US-11055887-B2
Application number	US-201816205010-A
Country	US
Kind code	B2
Filing date	Nov 29, 2018
Priority date	Nov 29, 2018
Publication date	Jul 6, 2021
Grant date	Jul 6, 2021

Abstract

Facial skin mask generated by a digital content creation system is described. The digital content creation system includes digital effects on skin in facial regions of digital content with efficiency and accuracy. Upon identifying a facial region within digital content, the system generates a first regional skin mask, a second regional skin mask, and combines both of the first and second regional skin masks to generate a facial skin mask indicative of skin of the identified facial regions depicted in digital content. The digital content creation system then modifies digital content by applying user selected digital effects to the skin of the facial region using the generated facial skin mask.

First claim

The invention claimed is:

1. In a digital medium environment, a method implemented by at least one computing device to apply a digital effect to digital content, the method comprising:
identifying a face in the digital content;
generating a first regional skin mask on a first region of the face and at least a second regional skin mask on a second region of the face;
generating, before the digital effect is applied, a facial skin mask by combining the first and second regional skin masks, the facial skin mask indicative of skin of the face;
receiving a user selection of the digital effect to be applied to the face after the first and second regional skin masks have been combined into the facial skin mask;
modifying the digital content by applying the digital effect on the skin of the face using the facial skin mask;
detecting a position change of the face within the digital content and comparing the position change to a predetermined threshold; and
based on the comparing:
generating an additional facial skin mask to use for applying the digital effect on the skin of the face if the position change exceeds the predetermined threshold; and
retaining the facial skin mask if the position change does not exceed the predetermined threshold.

2. The method as described in claim 1 , wherein the digital effect comprises at least one of digital paint, digital skin tone, digital lighting, or digital smoothing.

3. The method as described in claim 1 , wherein the face in the digital content is identified based on defined landmarks.

4. The method as described in claim 1 , wherein generating the first regional skin mask comprises:
extracting a subset of landmarks from facial landmarks identified of the face, the subset of landmarks defining the first region of the face, the first region corresponding to a forehead comprising hair interspersed with skin; and
segmenting the first region into a first segment corresponding to the hair and a second segment corresponding to the skin.

5. The method as described in claim 4 , wherein segmenting the first region into the first and second segments comprises:
identifying pixels associated with the subset of landmarks defining the first region; and
processing the pixels associated with the subset of landmarks defining the first region using k-means algorithm such that the pixels corresponding to the hair of the forehead and the pixels corresponding to the skin of the forehead are separated into the first segment and the second segment, respectively.

6. The method as described in claim 4 , wherein applying the digital effect to the skin of the face is effective to apply the digital effect to the second segment corresponding to the skin of the forehead.

7. The method as described in claim 1 , wherein generating the second regional skin mask comprises extracting a subset of landmarks from facial landmarks identified of the face, the subset of landmarks defining the second region of the face.

8. The method as described in claim 7 , further comprising tracing contours of the second region of the face based on the subset of landmarks defining the second region of the face, the contours used to identify portions of the face that correspond to the skin and portions of the face that correspond to excluded facial features.

9. The method as described in claim 8 , wherein applying the digital effect to the skin of the face using the facial skin mask is effective to apply the digital effect to the portions of the face that correspond to the skin without applying the digital effect to the portions of the face that correspond to the excluded facial features.

10. The method as described in claim 1 , further comprising applying the digital effect to the skin of the face using the additional facial skin mask.

11. In a digital medium environment, a system configured to apply a digital effect to digital content, the system comprising:
a facial region identification module implemented at least partially in hardware of a computing device to identify a face in the digital content;
a first regional skin mask generation module implemented at least partially in hardware of the computing device to generate a first regional skin mask on a first region of the face;
a second regional skin mask generation module implemented at least partially in hardware of the computing device to generate a second regional skin mask on a second region of the face;
a facial skin mask generation module implemented at least partially in hardware of the computing device to generate a facial skin mask by combining the first and second regional skin masks, the facial skin mask indicative of skin of the face;
a digital effect input module implemented at least partially in hardware of the computing device to receive a user selection of the digital effect to be applied to the face after the first and second regional skin masks have been combined into the facial skin mask;
the facial skin mask generation module further implemented to modify the digital content by applying the digital effect on the skin of the face using the facial skin mask;
the facial region identification module further implemented to detect a position change of the face within the digital content and compare the position change to a predetermined threshold; and
the facial skin mask generation module further implemented to generate an additional facial skin mask if the position change exceeds the predetermined threshold and retain the facial skin mask if the position change does not exceed the predetermined threshold.

12. The system as described in claim 11 , wherein the digital effect comprises at least one of digital paint, digital skin tone, digital lighting, or digital smoothing.

13. The system as described in claim 11 , wherein the face in the digital content is identified based on landmarks.

14. The system as described in claim 11 , wherein the first regional skin mask generation module generates the first regional skin mask by:
extracting a subset of landmarks from facial landmarks identified of the face, the subset of landmarks defining the first region of the face, the first region corresponding to a forehead comprising hair interspersed with skin; and
segmenting the first region into a first segment corresponding to the hair and a second segment corresponding to the skin.

15. The system as described in claim 14 , wherein the first regional skin mask generation module segments the first region into the first and second segments by:
identifying pixels associated with the subset of landmarks defining the first region; and
processing the pixels associated with the subset of landmarks defining the first region using k-means algorithm such that the pixels corresponding to the hair of the forehead and the pixels corresponding to the skin of the forehead are separated into the first segment and the second segment, respectively.

16. The system as described in claim 14 , wherein the facial skin mask generation module is further implemented to apply the digital effect to the skin of the face by applying the digital effect to the second segment corresponding to the skin of the forehead.

17. The system as described in claim 11 , wherein the second regional skin mask generation module generates the second regional skin mask by:
extracting a subset of landmarks from facial landmarks identified of the face, the subset of landmarks defining the second region of the face; and
tracing contours of the second region of the face based on the subset of landmarks defining the second region of the face, the contours used to iden

Assignees

• Adobe Inc

Inventors

• Jia Yumin
• Lu Xin
• Chien Jen-Chan

Classifications

• G06V40/162

  using pixel segmentation or colour matching · CPC title

• G06V40/168

  Feature extraction; Face representation · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06F3/04845 Primary

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title