Techniques for identifying skin color in images having uncontrolled lighting conditions

Patent metadata

Field	Value
Publication number	US-12437577-B2
Application number	US-202117538449-A
Country	US
Kind code	B2
Filing date	Nov 30, 2021
Priority date	Jul 18, 2019
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

In some embodiments of the present disclosure, one or more machine learning models are trained to accurately estimate skin color in one or more images regardless of the lighting conditions. In some embodiments, the models can then be used to estimate a skin color in a new image, and that estimated skin color can be used for a variety of purposes. For example, the skin color may be used to generate a recommendation for a foundation shade that accurately matches the skin color, or a recommendation for another cosmetic product that is complimentary with the estimated skin color. Thus, the need for an in-person test of the product is eliminated.

First claim

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method of training machine learning models to determine skin colors of faces under different illuminants, the method comprising:
receiving at least one training image that includes a face of a training subject;
receiving tagging information for the at least one training image;
adding the at least one training image and the tagging information to a training data store; and
training at least one machine learning model to determine skin colors of faces using the training data set;
wherein training the at least one machine learning model to determine skin colors of faces using the training data set includes:
training a first machine learning model that processes a training image as input to produce an indication of an illuminant color as an output; and
training a second machine learning model that processes the training image and the indication of the illuminant color as input to produce an indication of skin color as an output.

2. The method of claim 1 , further comprising normalizing the at least one training image to create at least one normalized training image.

3. The method of claim 2 , wherein normalizing the at least one training image to create at least one normalized training image includes:
detecting a face in the at least one training image;
centering the face in the at least one training image; and
zooming the at least one training image such that the face is a predetermined size.

4. The method of claim 1 , wherein receiving at least one training image includes:
receiving a video; and
extracting at least one training image from the video.

5. The method of claim 1 , wherein receiving tagging information for the at least one training image includes:
detecting a color reference chart in the training image; and
determining an illuminant color based on the color reference chart.

6. The method of claim 5 , further comprising adjusting a color of the at least one training image based on the determined illuminant color.

7. The method of claim 1 , wherein receiving tagging information for the at least one training image includes receiving skin color information collected from the training subject by a spectrophotometer.

8. A system, comprising:
a skin color estimation unit including computational circuitry configured to:
receive at least one training image that includes a face of a training subject;
receive tagging information for the at least one training image;
add the at least one training image and the tagging information to a training data store; and
train at least one machine learning model to determine skin colors of faces using the training data set;
wherein training the at least one machine learning model to determine skin colors of faces using the training data set includes:
training a first machine learning model that processes a training image as input to produce an indication of an illuminant color as an output; and
training a second machine learning model that processes the training image and the indication of the illuminant color as input to produce an indication of skin color as an output.

9. The system of claim 8 , wherein the computational circuitry is further configured to normalize the at least one training image to create at least one normalized training image.

10. The system of claim 9 , wherein normalizing the at least one training image to create at least one normalized training image includes:
detecting a face in the at least one training image;
centering the face in the at least one training image; and
zooming the at least one training image such that the face is a predetermined size.

11. The system of claim 8 , wherein receiving at least one training image includes:
receiving a video; and
extracting at least one training image from the video.

12. The system of claim 8 , wherein receiving tagging information for the at least one training image includes:
detecting a color reference chart in the training image; and
determining an illuminant color based on the color reference chart.

13. The system of claim 12 , wherein the computational circuitry is further configured to adjust a color of the at least one training image based on the determined illuminant color.

14. The system of claim 8 , wherein receiving tagging information for the at least one training image includes receiving skin color information collected from the training subject by a spectrophotometer.

15. A method of using one or more machine learning models to estimate a skin color of a face, the method comprising:
receiving, by a computing device, a video;
extracting, by the computing device, a plurality of images from the video that include a face of a live subject;
processing, by the computing device, the plurality of images using at least one machine learning model to obtain a plurality of determinations of a skin color of the face;
combining, by the computing device, the plurality of determinations of the skin color of the face to determine a combined determination of the skin color of the face; and
presenting, by the computing device, the combined determination of the skin color of the face.

16. The method of claim 15 , further comprising normalizing, by the computing device, the plurality of images to create a plurality of normalized images.

17. The method of claim 16 , wherein normalizing the plurality of images to create a plurality of normalized images includes, for each image:
detecting, by the computing device, a face in the image;
centering, by the computing device, the face in the image; and
zooming, by the computing device, the image such that the face is a predetermined size.

18. The method of claim 15 , wherein processing the plurality of images using at least one machine learning model to obtain a plurality of determinations of a skin color includes, for each image:
executing, by the computing device, a first machine learning model using the image as input to generate an indication of a lighting condition as output; and
executing, by the computing device, a second machine learning model using the image and the indication of the lighting condition as input to generate the determination of the skin color.

19. The method of claim 15 , further comprising:
executing, by the computing device, a recommender model using the combined determination of the skin color as input to generate a product recommendation.

Assignees

• Oreal

Inventors

• Elfakhri Christine
• VALCESCHINI Florent
• Tran Loic
• Perrot Matthieu
• Kips Robin
• Malherbe Emmanuel

Classifications

• G06V10/56

  relating to colour · CPC title

• G01J3/46

  Measurement of colour; Colour measuring devices, e.g. colorimeters (measuring colour temperature G01J5/60) · CPC title

• B44D3/003

  Methods or devices for colour determination, selection or synthesis, e.g. use of colour tables · CPC title

• A61B5/441

  Skin evaluation, e.g. for skin disorder diagnosis · CPC title

• A61B5/0077

  Devices for viewing the surface of the body, e.g. camera, magnifying lens · CPC title