Multi-camera video capture for accurate skin tone measurement

Patent metadata

Field	Value
Publication number	US-12394098-B2
Application number	US-202318154669-A
Country	US
Kind code	B2
Filing date	Jan 13, 2023
Priority date	Jan 13, 2023
Publication date	Aug 19, 2025
Grant date	Aug 19, 2025

Abstract

A computing device obtains a first video recorded by a first camera (e.g., a rear-facing camera of a mobile computing device) having a first field of view in an illuminated environment and obtains a second video recorded by a second camera having a second field of view (e.g., a forward-facing camera of a mobile computing device) in the illuminated environment that differs from the first field of view. The second field of view is directed toward a face of a live subject. The computing device extracts lighting environment images from the first video and face images from the second video. The computing device processes the face images and the lighting environment images (e.g., using machine learning models) to obtain a plurality of determinations of skin tone of the face and combines the determinations of skin tone of the face to determine a combined determination of skin tone.

First claim

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method of estimating skin tone based on face images, the method comprising:
obtaining, by a computing device, a first video recorded by a first camera having a first field of view in an illuminated environment;
obtaining, by the computing device, a second video recorded by a second camera having a second field of view in the illuminated environment that differs from the first field of view, wherein the second field of view is directed toward a face of a live subject;
extracting, by the computing device, a plurality of lighting environment images from the first video;
extracting, by the computing device, a plurality of face images from the second video that include the face of the live subject;
processing, by the computing device, the face images and the lighting environment images to obtain a plurality of determinations of skin tone of the face;
combining, by the computing device, the determinations of skin tone of the face to determine a combined determination of skin tone; and
presenting, by the computing device, the combined determination of skin tone.

2. The method of claim 1 further comprising constructing a panoramic image from the first video.

3. The method of claim 1 , wherein processing the face images and the lighting environment images includes executing at least one machine learning model using the face images and the lighting environment images as input to generate the determinations of skin tone of the face as output.

4. The method of claim 3 wherein executing the at least one machine learning model includes:
executing a first machine learning model using the face images and the lighting environment images as input to generate lighting condition information for each of the face images as output; and
executing a second machine learning model using the face images and the corresponding lighting condition information as input to generate the determinations of skin tone of the face as output.

5. The method of claim 4 , wherein the lighting condition information comprises an illuminant color.

6. The method of claim 4 , wherein the determinations of skin tone each include a corresponding confidence level.

7. The method of claim 6 further comprising:
comparing the corresponding confidence level with a threshold confidence level; and
omitting the determination of skin tone from the combining step where the corresponding confidence level is less than the threshold confidence level.

8. The method of claim 1 , wherein extracting the face images from the second video comprises, for individual frames of the second video:
measuring a brightness level of the frame; and
omitting the frame from the extracted face images where the measured brightness level is less than a minimum threshold brightness level or greater than a maximum threshold brightness level.

9. The method of claim 1 , wherein the first video is recorded at the same time as the second video.

10. A system, comprising:
a skin tone estimation unit including computational circuitry configured to;
obtain a first video recorded by a first camera having a first field of view in an illuminated environment;
obtain a second video recorded by a second camera having a second field of view in the illuminated environment that differs from the first field of view, wherein the second field of view is directed toward a face of a live subject;
extract a plurality of lighting environment images from the first video;
extract a plurality of face images from the second video that include the face of the live subject;
process the face images and the lighting environment images to obtain a plurality of determinations of skin tone of the face;
combine the determinations of skin tone of the face to determine a combined determination of skin tone of the face; and
present the combined determination of the skin tone of the face.

11. The system of claim 10 , wherein the first camera is a rear-facing camera, and wherein the computational circuitry is further configured to construct a panoramic image from the first video.

12. The system of claim 10 , wherein the computational circuitry is further configured to execute at least one machine learning model using the face images and the lighting environment images as input to generate the determinations of skin tone of the face as output.

13. The system of claim 10 , wherein the computational circuitry is further configured to:
execute a first machine learning model using the face images and the lighting environment images as input to generate lighting condition information for each of the face images as output; and
execute a second machine learning model using the face images and the corresponding lighting condition information as input to generate the determinations of skin tone of the face as output.

14. The system of claim 13 , wherein the lighting condition information comprises an illuminant color.

15. The system of claim 13 , wherein the determinations of skin tone each include a corresponding confidence level.

16. The system of claim 15 , wherein the computational circuitry is further configured to combine the determinations of skin tone of the face to determine a combined determination of skin tone by, for each of the determinations of skin tone:
comparing the corresponding confidence level with a threshold confidence level; and
omitting the determination of skin tone from the combining step where the corresponding confidence level is less than the threshold confidence level.

17. The system of claim 10 , wherein the computational circuitry is further configured to, for individual frames of the second video:
measure a brightness level of the frame; and
omit the frame from the extracted face images where the measured brightness level is less than a minimum threshold brightness level or greater than a maximum threshold brightness level.

18. A non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by one or more processors of a computer system, cause the computer system to perform actions comprising:
presenting a user interface on a mobile computing device instructing a user to capture a panoramic image in an illuminated environment;
responsive to user input via the user interface, recording a first video with a rear-facing camera of the mobile computing device in the illuminated environment and recording a second video with a front-facing camera of the mobile computing device in the illuminated environment;
extracting a plurality of lighting environment images from the first video;
extracting a plurality of face images from the second video that include the user's face;
processing the face images and the lighting environment images to obtain a plurality of determinations of a skin tone of the face; and
combining the plurality of determinations of the skin tone of the face to determine a combined determination of the skin tone of the face.

19. The non-transitory computer-readable medium of claim 18 , wherein processing the face images and the lighting environment images includes:
executing, by the computing device, a first machine learning model using the face images and the lighting environment images as input to generate lighting condition information for each of the face images as output; and
executing, by the computing device, a second machine learning model using the face images and the corresponding lighting condit

Assignees

• Oreal

Inventors

• Charraud Grégoire
• Lee Ji
• Bansal Shivya

Classifications

• H04N5/77

  between a recording apparatus and a television camera · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• G06V10/56

  relating to colour · CPC title

• G06V40/169 Primary

  Holistic features and representations, i.e. based on the facial image taken as a whole · CPC title