Generating a product recommendation based on a user reaction

Patent metadata

Field	Value
Publication number	US-10475106-B1
Application number	US-201916249062-A
Country	US
Kind code	B1
Filing date	Jan 16, 2019
Priority date	Jan 16, 2019
Publication date	Nov 12, 2019
Grant date	Nov 12, 2019

Abstract

A device may process, after obtaining an image that includes image data concerning a product, first audio data obtained concerning a first utterance of a user of a user device, and first video data obtained concerning a first eye gaze direction of the user, to determine a first reaction of the user to the image. The device may process, after causing display of the image and an overlay superimposed on the image by the user device, second audio data obtained concerning a second utterance of the user, and second video data obtained concerning a second eye gaze direction of the user, to determine a second reaction of the user to the image and the overlay. The device may retrain a product recommendation model using the first reaction and the second reaction, and may generate and cause display of a product recommendation based on the product recommendation model.

First claim

What is claimed is:

1. A method, comprising:
obtaining, via a first camera of a mobile device, image data concerning a product;
obtaining, by the mobile device and while obtaining the image data, first audio data concerning a first utterance of a user of the mobile device;
obtaining, by a second camera of the mobile device and while obtaining the image data, first video data concerning a first facial expression of the user,
the first camera facing a different direction than the second camera;

processing, by the mobile device, the first audio data and the first video data to cause a first reaction of the user to the product to be determined;
causing, by the mobile device and after obtaining the image data, display of an augmented reality (AR) overlay superimposed on the image data by the mobile device;
obtaining, by the mobile device and after causing display of the AR overlay, second audio data concerning a second utterance of the user;
obtaining, by the second camera of the mobile device and after causing display of the AR overlay, second video data concerning a second facial expression of the user;
processing, by the mobile device, the second audio data and the second video data to cause a second reaction of the user to the AR overlay to be determined;
causing, by the mobile device, a product recommendation model to be updated based on the first reaction and the second reaction; and
causing, by the mobile device, display of a product recommendation generated based on the product recommendation model.

2. The method of claim 1 , wherein causing display of the image data and the augmented reality (AR) overlay comprises:
processing the image data to cause product information to be determined; and
generating the AR overlay based on the product information.

3. The method of claim 1 , wherein obtaining the first audio data concerning the first utterance of the user of the mobile device comprises:
obtaining the first audio data using a microphone of the mobile device,
wherein obtaining the second audio data concerning the second utterance of the user comprises:
obtaining the second audio data using the microphone of the mobile device.

4. The method of claim 1 , wherein the product is a car,
wherein the AR overlay includes information that indicates at least one of:
a make of the car;
a model of the car;
a model year of the car;
a price of the car;
a safety feature of the car;
a condition of the car;
a mileage of the car;
a transmission type of the car;
a seller of the car;
an ownership history of the car;
an accident report for the car; or
an availability of the car.

5. The method of claim 1 , wherein causing the product recommendation model to be updated based on the first reaction and the second reaction comprises:
causing the product recommendation model to be updated using a machine learning technique.

6. The method of claim 1 , wherein processing the first audio data and the first video data to cause the first reaction of the user to the product to be determined comprises:
processing the first audio data using a natural language processing technique to identify the first utterance;
processing the first video data using a facial analysis technique to identify the first facial expression; and
causing the first reaction to be determined based on the first utterance and the first facial expression.

7. The method of claim 6 , wherein causing the first reaction to be determined based on the first utterance and the first facial expression comprises:
processing the first utterance and the first facial expression using a sentiment analysis technique.

8. The method of claim 1 , wherein processing the second audio data and the second video data to cause the second reaction of the user to the AR overlay to be determined comprises:
processing the second audio data using a speech recognition technique to identify one or more words spoken by the user;
processing the second video data using an emotion recognition technique to identify an emotion of the user; and
causing the second reaction to be determined based on the one or more words and the emotion.

9. The method of claim 1 , further comprising:
processing, using a sentiment analysis technique, the first utterance of the user of the mobile device to categorize the first utterance as:
a positive utterance,
a neutral utterance, or
a negative utterance.

10. A mobile device, comprising:
one or more memories; and
one or more processors, communicatively coupled to the one or more memories, to:
obtain, via a first camera of the mobile device, an image that includes image data concerning a product;
prompt a user of the mobile device to comment on the product;
obtain, while prompting the user to comment on the product, first audio data concerning a first utterance of the user;
obtain, from a second camera and while prompting the user to comment on the product, first video data concerning a first facial expression of the user,
the first camera facing a different direction than the second camera;

process the first audio data and the first video data to cause a first reaction of the user to the product to be determined;
cause, after obtaining the image, display of the image and an augmented reality (AR) overlay superimposed on the image by the mobile device;
prompt the user to comment on the display of the image and the AR overlay;
obtain, while prompting the user to comment on the display of the image and the AR overlay, second audio data concerning a second utterance of the user;
obtain, from the second camera of the mobile while prompting the user to comment on the display of the image and the AR overlay, second video data concerning a second facial expression of the user;
process the second audio data and the second video data to cause a second reaction of the user to the image and the AR overlay to be determined;
cause a product recommendation model to be trained using the first reaction and the second reaction; and
send a product recommendation profile generated based on the product recommendation model to a remote device.

11. The mobile device of claim 10 , wherein the one or more processors, when prompting the user of the mobile device to comment on the product, are to:
cause display by the mobile device of a message asking the user to talk about the product.

12. The mobile device of claim 10 , wherein the one or more processors, when prompting the user to comment on the display of the image and the AR overlay, are to:
cause the mobile device to display a message that instructs the user to discuss the image and the AR overlay.

13. The mobile device of claim 10 , wherein the product has a particular product type,
wherein the product recommendation profile includes information concerning the particular product type.

14. The mobile device of claim 10 , wherein the product has a particular product type,
wherein the product recommendation profile includes information concerning recommendations for a different product that has a different product type than the particular product type.

15. The mobile device of claim 10 , wherein the one or more processors are further to:
receive a message from the remote device,
wherein the message was generated by the remote device based on the product recommendation profile,
wherein the message includes a recommendation for a related product.

16. A non-transitory computer-readable medium storing instructions, the instructions comprising:
one or more instructions that, when executed by one or more processors of a mobil

Assignees

• Capital One Services Llc

Inventors

• Tang Qiaochu
• Hoover Jason
• Price Micah
• Dagley Geoffrey
• Wylie Stephen

Classifications

• G06N20/10

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• H04M2250/52

  including functional features of a camera · CPC title

• H04M1/6008

  in the transmitter circuit · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title