Systems and methods for providing personalized product recommendations using deep learning

Patent metadata

Field	Value
Publication number	US-11521013-B2
Application number	US-202117317742-A
Country	US
Kind code	B2
Filing date	May 11, 2021
Priority date	Oct 25, 2018
Publication date	Dec 6, 2022
Grant date	Dec 6, 2022

Abstract

Embodiments of the present disclosure provide systems, methods, and computer-readable media that use machine learning models to enable computing devices to detect and identify cosmetic products in face images. In some embodiments, a model training system may gather training data for building the machine learning models by analyzing face images associated with tagging data. In some embodiments, a recommendation system may be configured to use the machine learning models generated by the model training system to detect products in face images, and to add information based on the detected products to a look data store, and/or to provide recommendations for similar looks from the look data store based on the detected products.

First claim

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method of using one or more machine learning models to process face images, the method comprising:
using, by a computing device, the one or more machine learning models to determine identities of one or more cosmetic products in a face image;
determining, by the computing device, one or more recommended looks based on the identities of the one or more cosmetic products in the face image;
determining, by the computing device, one or more cosmetic products associated with the one or more recommended looks; and
presenting, by the computing device, the one or more recommended looks or the one or more cosmetic products to a user;
wherein the determining one or more recommended looks based on the identities of the one or more cosmetic products in the face image includes:
adding a new look that includes the identities of the one or more cosmetic products to a look data store;
performing a cluster analysis on looks stored in the look data store; and
determining the one or more recommended looks based on other looks included in a cluster with the new look.

2. The method of claim 1 , wherein the one or more machine learning models include one or more convolutional neural networks.

3. The method of claim 1 , wherein the face image is included in a social media post.

4. The method of claim 3 , further comprising:
determining, by the computing device, a look based on the determined identities of the one or more cosmetic products;
determining, by the computing device, a sentiment score based on the social media post; and
storing, by the computing device, the sentiment score and the look in a look data store.

5. The method of claim 1 , wherein using the one or more machine learning models to determine the identities of the one or more cosmetic products in the face image includes:
extracting a portion of the face image associated with a predetermined facial feature; and
providing the portion of the face image to a machine learning model trained to determine identities of cosmetic products applied to the predetermined facial feature.

6. The method of claim 5 , wherein the predetermined facial feature is an eye, lips, a forehead, a cheekbone, or a nose.

7. The method of claim 1 , wherein determining the one or more recommended looks based on the identities of the one or more cosmetic products in the face image includes:
determining one or more looks that include at least one cosmetic product with a color or a texture that matches a color or a texture of the identified one or more cosmetic products in the face image, and at least one product that does not match the color or the texture of the identified one or more cosmetic products in the face image.

8. The method of claim 1 , wherein the one or more machine learning models are trained by:
extracting, by a training computing device, a training face image from a training social media post;
determining, by the training computing device, an identity of a product present in the training face image based on non-image data by performing actions including examining text associated with the training social media post outside of the training face image and determining the identity of the product based on the text;
extracting, by the training computing device, a portion of the training face image that includes a facial feature associated with the product;
adding, by the training computing device, the portion of the training face image and the identity of the product to a training data set; and
training, by the training computing device, the machine learning model to identify the product in face images using the training data set.

9. The method of claim 8 , wherein the text associated with the training social media post outside of the training face image includes semantically demarcated content.

10. The method of claim 9 , wherein the semantically demarcated content is a hashtag.

11. A system for using one or more machine learning models to process face images, the system comprising:
circuitry for using the one or more machine learning models to determine identities of one or more cosmetic products in a face image;
circuitry for determining one or more recommended looks based on the identities of the one or more cosmetic products in the face image;
circuitry for determining one or more cosmetic products associated with the one or more recommended looks; and
circuitry for presenting the one or more recommended looks or the one or more cosmetic products to a user;
wherein the determining one or more recommended looks based on the identities of the one or more cosmetic products in the face image includes:
adding a new look that includes the identities of the one or more cosmetic products to a look data store;
performing a cluster analysis on looks stored in the look data store; and
determining the one or more recommended looks based on other looks included in a cluster with the new look.

12. The system of claim 11 , wherein the one or more machine learning models include one or more convolutional neural networks.

13. The system of claim 11 , wherein the face image is included in a social media post.

14. The system of claim 13 , further comprising:
circuitry for determining a look based on the determined identities of the one or more cosmetic products;
circuitry for determining a sentiment score based on the social media post; and
circuitry for storing the sentiment score and the look in a look data store.

15. The system of claim 11 , wherein using the one or more machine learning models to determine the identities of the one or more cosmetic products in the face image includes:
extracting a portion of the face image associated with a predetermined facial feature; and
providing the portion of the face image to a machine learning model trained to determine identities of cosmetic products applied to the predetermined facial feature.

16. The system of claim 11 , wherein determining the one or more recommended looks based on the identities of the one or more cosmetic products in the face image includes:
determining one or more looks that include at least one cosmetic product with a color or a texture that matches a color or a texture of the identified one or more cosmetic products in the face image, and at least one product that does not match the color or the texture of the identified one or more cosmetic products in the face image.

17. The system of claim 11 , wherein the one or more machine learning models are trained by:
extracting, by a training computing device, a training face image from a training social media post;
determining, by the training computing device, an identity of a product present in the training face image based on non-image data by performing actions including examining text associated with the training social media post outside of the training face image and determining the identity of the product based on the text;
extracting, by the training computing device, a portion of the training face image that includes a facial feature associated with the product;
adding, by the training computing device, the portion of the training face image and the identity of the product to a training data set; and
training, by the training computing device, the machine learning model to identify the product in face images using the training data set.

18. The system of claim 17 , wherein the text associated with the training social media post outside of the training

Assignees

• Oreal

Inventors

• Charraud Grégoire
• Malaprade Helga
• Thiebaut Géraldine
• Perrot Matthieu
• Kips Robin

Classifications

• G06V40/16 Primary

  Human faces, e.g. facial parts, sketches or expressions · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title