Skin tone assisted digital image color matching

Patent metadata

Field	Value
Publication number	US-11610433-B2
Application number	US-202117154830-A
Country	US
Kind code	B2
Filing date	Jan 21, 2021
Priority date	Oct 4, 2019
Publication date	Mar 21, 2023
Grant date	Mar 21, 2023

Abstract

In implementations of skin tone assisted digital image color matching, a device implements a color editing system, which includes a facial detection module to detect faces in an input image and in a reference image, and includes a skin tone model to determine a skin tone value reflective of a skin tone of each of the faces. A color matching module can be implemented to group the faces into one or more face groups based on the skin tone value of each of the faces, match a face group pair as an input image face group paired with a reference image face group, and generate a modified image from the input image based on color features of the reference image, the color features including face skin tones of the respective faces in the face group pair as part of the color features applied to modify the input image.

First claim

The invention claimed is:

1. A method implemented by a computing device, the method comprising:
receiving an input image to modify based on color features of a reference image;
detecting faces in the input image and in the reference image;
determining a skin tone value reflective of a skin tone of each of the faces;
grouping the faces into one or more face groups based on the skin tone value of each of the faces in the input image and in the reference image, the one or more face groups including an input image face group and a reference image face group; and
generating a modified image from the input image based on the color features of the reference image, the color features applied to modify the input image including face skin tones of the faces in the reference image face group, yet maintaining the skin tone value of each of the faces from the input image in the modified image.

2. The method of claim 1 , wherein the input image and the reference image are one of digital images or digital video frames.

3. The method of claim 1 , wherein the grouping the faces into the one or more face groups comprises:
grouping input image faces into one or more input image face groups, the input image face group including one or more of the input image faces having skin tone value differences less than or equal to a grouping threshold; and
grouping reference image faces into one or more reference image face groups, the reference image face group including one or more of the reference image faces having skin tone value differences less than or equal to the grouping threshold.

4. The method of claim 3 , further comprising:
calculating an average skin tone value from the skin tone values of respective one or more faces in each of the one or more input image face groups and in each of the one or more reference image face groups.

5. The method of claim 4 , further comprising:
matching a face group pair as the input image face group paired with the reference image face group, wherein the matching the face group pair comprises:
determining the face group pair based on a matching criteria of the face group pair having a difference of the respective average skin tone values being less than or equal to a matching threshold;
identifying input image regions of the input image and reference image regions of the reference image that include the faces;
calculating area sums of the input image regions and the reference image regions corresponding to the faces grouped in respective one or more input image face groups and respective one or more reference image face groups; and
matching the face group pair as the input image face group having a largest possible area sum of the input image regions that meets the matching criteria and the reference image face group having a largest possible area sum of the reference image regions that meets the matching criteria.

6. The method of claim 5 , wherein the generating the modified image includes color matching midtone color features of the reference image applied to the input image, the color matching including using the face skin tones of the faces in the face group pair as part of the midtone color features.

7. The method of claim 6 , wherein the color matching the midtone color features of the reference image applied to the input image is effective to maintain original captured image color of the faces in the input image.

8. The method of claim 1 , wherein the determining the skin tone value comprises:
inputting image regions of the input image and the reference image that include the faces into a skin tone model; and
utilizing the skin tone model that determines the skin tone values of the respective faces based on at least skin tone identifiers and lighting features corresponding to the image regions.

9. A device comprising:
a memory to maintain an input image and a reference image as one of digital images or digital video frames; and
a processor system configured to implement a color editing system at least partially in hardware, the color editing system including:
a facial detection module implemented to detect one or more faces in the input image and in the reference image;
a skin tone model implemented to determine a skin tone value reflective of a skin tone of each of the one or more faces; and
a color matching module implemented to:
group the one or more faces into one or more face groups based on the skin tone value of each of the one or more faces in the input image and in the reference image, the one or more face groups including an input image face group and a reference image face group; and
generate a modified image from the input image based on color features of the reference image, the color features applied to modify the input image including face skin tones of the faces in the reference image face group, yet maintain the skin tone value of each of the faces from the input image in the modified image.

10. The device of claim 9 , wherein, to group the one or more faces into the one or more face groups, the color matching module is implemented to:
group input image faces into one or more input image face groups, the input image face group including one or more of the input image faces having skin tone value differences less than or equal to a grouping threshold; and
group reference image faces into one or more reference image face groups, the reference image face group including one or more of the reference image faces having skin tone value differences less than or equal to the grouping threshold.

11. The device of claim 10 , wherein the color matching module is implemented to calculate an average skin tone value from the skin tone values of respective one or more faces in each of the one or more input image face groups and the one or more reference image face groups.

12. The device of claim 11 , wherein the color matching module is implemented to:
match a face group pair as the input image face group paired with the reference image face group based on a matching criteria of the face group pair having a difference of the respective average skin tone values being less than or equal to a matching threshold; and
match the face group pair comprising to:
identify input image regions of the input image and reference image regions of the reference image that include the faces;
calculate area sums of the input image regions and the reference image regions corresponding to the one or more faces grouped in respective one or more input image face groups and respective one or more reference image face groups; and
match the face group pair as the input image face group having a largest possible area sum of the input image regions that meets the matching criteria and the reference image face group having a largest possible area sum of the reference image regions that meets the matching criteria.

13. The device of claim 12 , wherein the color matching module of the color editing system is implemented to color match midtone color features of the reference image applied to the input image to generate the modified image using the face skin tones of the faces in the face group pair as part of the midtone color features.

14. The device of claim 13 , wherein the color match by the color matching module is effective to maintain original captured image color of the one or more faces in the input image.

15. The device of claim 9 , wherein the skin tone model of the color editing system is implemented to determine the skin tone values of respective one or more faces based on at least skin tone identifiers and lighting features corresponding to image regions of the input image a

Assignees

• Adobe Inc

Inventors

• Sethi Kartik
• Wang Oliver
• Mohandoss Tharun
• Shechtman Elya
• Nanda Chetan

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06T2207/10024

  Color image · CPC title

• G06V40/162 Primary

  using pixel segmentation or colour matching · CPC title

• G06T7/90 Primary

  Determination of colour characteristics · CPC title