Generating a product recommendation based on a user reaction

Patent metadata

Field	Value
Publication number	US-11610250-B2
Application number	US-202017247494-A
Country	US
Kind code	B2
Filing date	Dec 14, 2020
Priority date	Jan 16, 2019
Publication date	Mar 21, 2023
Grant date	Mar 21, 2023

Abstract

A device may process, after obtaining an image that includes image data concerning a product, first audio data obtained concerning a first utterance of a user of a user device, and first video data obtained concerning a first eye gaze direction of the user, to determine a first reaction of the user to the image. The device may process, after causing display of the image and an overlay superimposed on the image by the user device, second audio data obtained concerning a second utterance of the user, and second video data obtained concerning a second eye gaze direction of the user, to determine a second reaction of the user to the image and the overlay. The device may retrain a product recommendation model using the first reaction and the second reaction, and may generate and cause display of a product recommendation based on the product recommendation model.

First claim

What is claimed is:

1. A method, comprising:
obtaining, via a camera of a mobile device, image data concerning a product;
causing, by the mobile device and after obtaining the image data, display of an overlay superimposed on the image data,
wherein the overlay includes product information associated with the image data;

obtaining, by the mobile device and while displaying the overlay superimposed on the image data, audio data of an utterance of a user of the mobile device;
processing the audio data to cause a reaction of the user to at least a portion of the product information to be determined; and
causing, by the mobile device, display of a product recommendation generated based on the reaction.

2. The method of claim 1 , wherein the reaction is determined based on processing the utterance using a sentiment analysis technique to categorize the utterance into a sentiment category.

3. The method of claim 2 , wherein the sentiment category includes at least one of a positive utterance, a neutral utterance, or a negative utterance.

4. The method of claim 1 , wherein the reaction is determined based on one or more words of the utterance.

5. The method of claim 1 , further comprising:
obtaining, by the mobile device and from another device, the product information based on processing the image data.

6. The method of claim 1 , further comprising:
determining the utterance concerns the reaction of the user to at least the portion of the product information.

7. The method of claim 1 , further comprising:
providing a prompt to comment on the product information,
the audio data being obtained based on providing the prompt.

8. A mobile device, comprising:
one or more memories; and
one or more processors, communicatively coupled to the one or more memories, to:
obtain, via a camera, image data concerning a product;
cause, after obtaining the image data, display of an overlay superimposed on the image data,
wherein the overlay includes product information associated with the image data;

obtain, while displaying the overlay superimposed on the image data, audio data of an utterance of a user of the mobile device;
process the audio data to cause a reaction of the user to at least a portion of the product information to be determined; and
cause display of a product recommendation generated based on the reaction.

9. The mobile device of claim 8 , wherein the reaction is determined based on processing the utterance using a sentiment analysis technique to categorize the utterance into a sentiment category.

10. The mobile device of claim 9 , wherein the sentiment category includes at least one of a positive utterance, a neutral utterance, or a negative utterance.

11. The mobile device of claim 8 , wherein the reaction is determined based on one or more words of the utterance.

12. The mobile device of claim 8 , wherein the one or more processors are further to:
obtain, from another device, the product information based on processing the image data.

13. The mobile device of claim 8 , wherein the one or more processors are further to:
determine the utterance concerns the reaction of the user to at least the portion of the product information.

14. The mobile device of claim 8 , wherein the one or more processors are further to:
provide a prompt to comment on the product information,
the audio data being obtained based on providing the prompt.

15. A non-transitory computer-readable medium storing instructions, the instructions comprising:
one or more instructions that, when executed by one or more processors of a mobile device, cause the one or more processors to:
obtain, via a camera, image data concerning a product;
cause, after obtaining the image data, display of an overlay superimposed on the image data,
wherein the overlay includes product information associated with the image data;

obtain, while displaying the overlay superimposed on the image data, audio data of an utterance of a user of the mobile device;
process the audio data to cause a reaction of the user to at least a portion of the product information to be determined; and
cause display of a product recommendation generated based on the reaction.

16. The non-transitory computer-readable medium of claim 15 , wherein the reaction is determined based on processing the utterance using a sentiment analysis technique to categorize the utterance into a sentiment category.

17. The non-transitory computer-readable medium of claim 16 , wherein the sentiment category includes at least one of a positive utterance, a neutral utterance, or a negative utterance.

18. The non-transitory computer-readable medium of claim 15 , wherein the reaction is determined based on one or more words of the utterance.

19. The non-transitory computer-readable medium of claim 15 , wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to:
obtain, from another device, the product information based on processing the image data.

20. The non-transitory computer-readable medium of claim 15 , wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to:
determine the utterance concerns the reaction of the user to at least the portion of the product information.

Assignees

• Capital One Services Llc

Inventors

• Tang Qiaochu
• Hoover Jason
• Price Micah
• Dagley Geoffrey
• Wylie Stephen

Classifications

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• H04M1/72406

  by software upgrading or downloading · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F16/54

  Browsing; Visualisation therefor · CPC title

• H04M1/72454

  according to context-related or environment-related conditions · CPC title