Systems and methods for providing personalized product recommendations using deep learning

Patent metadata

Field	Value
Publication number	US-11010636-B2
Application number	US-201816171153-A
Country	US
Kind code	B2
Filing date	Oct 25, 2018
Priority date	Oct 25, 2018
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

Embodiments of the present disclosure provide systems, methods, and computer-readable media that use machine learning models to enable computing devices to detect and identify cosmetic products in face images. In some embodiments, a model training system may gather training data for building the machine learning models by analyzing face images associated with tagging data. In some embodiments, a recommendation system may be configured to use the machine learning models generated by the model training system to detect products in face images, and to add information based on the detected products to a look data store, and/or to provide recommendations for similar looks from the look data store based on the detected products.

First claim

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method of training a machine learning model to detect cosmetic products in a face image, the method comprising:
extracting, by a computing device, the face image from a social media post;
determining, by the computing device, a product present in the face image based on non-image data by performing actions including:
examining text associated with the social media post outside of the face image; and
determining the product based on the text;

extracting, by the computing device, a portion of the face image that includes a facial feature associated with the product;
adding, by the computing device, the portion of the face image and an identification of the product to a training data set; and
training, by the computing device, the machine learning model to detect the product in face images using the training data set.

2. The method of claim 1 , wherein the text associated with the social media post outside of the face image includes semantically demarcated content.

3. The method of claim 2 , wherein
the semantically demarcated content is a hashtag.

4. The method of claim 1 , wherein determining the product present in the face image includes:
extracting a portion of the face image that includes a predetermined facial feature;
detecting a color and a texture included in the portion of the face image; and
determining a product associated with the detected color and texture based on laboratory measurements of the product.

5. The method of claim 4 , wherein the predetermined facial feature is lips.

6. The method of claim 1 , wherein the machine learning model is a convolutional neural network.

7. The method of claim 1 , wherein extracting the portion of the face image that includes the facial feature associated with the product includes using a computer vision technique.

8. A system for training a machine learning model to detect cosmetic products in a face image, the system comprising:
circuitry for determining a product present in the face image based on non-image data;
circuitry for extracting a portion of the face image that includes a facial feature associated with the product;
circuitry for adding the portion of the face image and an identification of the product to a training data set; and
circuitry for training the machine learning model to detect the product in face images using the training data set;
wherein determining the product present in the face image includes:
extracting a portion of the face image that includes a predetermined facial feature;
detecting a color and a texture included in the portion of the face image; and
determining a product associated with the detected color and texture based on laboratory measurements of the product.

9. The system of claim 8 , further comprising circuitry for extracting the face image from a social media post.

10. The system of claim 9 , wherein determining the product present in the face image based on non-image data includes:
examining text associated with the social media post; and
determining the product based on the text.

11. The system of claim 8 , wherein the machine learning model is a convolutional neural network.

12. A non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by one or more processors of a computing device, cause the computing device to perform actions for training a machine learning model to detect cosmetic products in a face image, the actions comprising:
extracting, by the computing device, the face image from a social media post;
determining, by the computing device, a product present in the face image based on non-image data by performing actions including:
examining text associated with the social media post outside of the face image; and
determining the product based on the text;

extracting, by the computing device, a portion of the face image that includes a facial feature associated with the product;
adding, by the computing device, the portion of the face image and an identification of the product to a training data set; and
training, by the computing device, the machine learning model to detect the product in face images using the training data set.

13. The non-transitory computer-readable medium of claim 11 , wherein the text associated with the social media post outside of the face image includes semantically demarcated content.

14. The non-transitory computer-readable medium of 13 , wherein the semantically demarcated content is a hashtag.

15. The non-transitory computer-readable medium of claim 12 , wherein determining the product present in the face image includes: extracting a portion of the face image that includes a predetermined facial feature; detecting a color and a texture included in the portion of the face image; and determining a product associated with the detected color and texture based on laboratory measurements of the product.

16. The non-transitory computer-readable medium of claim 15 , wherein the predetermined facial feature is lips.

17. The non-transitory computer-readable medium of claim 12 , wherein the machine learning model is a convolutional neural network.

18. The non-transitory computer-readable medium of claim 12 , wherein extracting the portion of the face image that includes the facial feature associated with the product includes using a computer vision technique.

Assignees

• Oreal

Inventors

• Charraud Grégoire
• Malaprade Helga
• Thiebaut Géraldine
• Perrot Matthieu
• Kips Robin

Classifications

• G06V40/16 Primary

  Human faces, e.g. facial parts, sketches or expressions · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title