Automatic response suggestions based on images received in messaging applications

Patent metadata

Field	Value
Publication number	US-10862836-B2
Application number	US-201916560815-A
Country	US
Kind code	B2
Filing date	Sep 4, 2019
Priority date	Sep 20, 2016
Publication date	Dec 8, 2020
Grant date	Dec 8, 2020

Abstract

Implementations relate to automatic response suggestions based on images received in messaging applications. In some implementations, a computer-executed method includes detecting a first image included within a first message received at a second device over a communication network from a first device of a first user, and programmatically analyzing the first image to extract a first image content. The method includes retrieving a first semantic concept associated with the first image content, programmatically generating a suggested response to the first message based on the first semantic concept, and transmitting instructions causing rendering of the suggested response in the messaging application as a suggestion to a second user of the second device.

First claim

What is claimed is:

1. A computer-executed method to automatically provide suggestions in a messaging application, the computer-executed method comprising:
receiving one or more messages including a first image and a second image at a second device from a first device over a communication network;
programmatically analyzing the first image to extract a first image content;
retrieving a first semantic concept associated with the first image content;
programmatically analyzing the second image to extract a second image content;
retrieving a second semantic concept associated with the second image content;
programmatically generating a suggested response to the one or more messages, wherein the suggested response is based on the first semantic concept and the second semantic concept; and
transmitting a command to cause rendering of the suggested response in the messaging application on the second device.

2. The computer-executed method of claim 1 , wherein the first semantic concept includes first text and the second semantic concept includes second text, and wherein the suggested response includes at least a portion of the first text from the first semantic concept and at least a portion of the second text from the second semantic concept.

3. The computer-executed method of claim 1 , wherein the suggested response includes a suggested image, wherein the suggested image depicts a first feature related to the first semantic concept and a second feature related to the second semantic concept.

4. The computer-executed method of claim 1 , further comprising:
upon receiving a selection of the suggested response based on input received at the second device, transmitting the suggested response over the communication network to the first device.

5. The computer-executed method of claim 1 , further comprising:
detecting a first textual content within the one or more messages, wherein programmatically generating the suggested response is further based on the first textual content of the one or more messages.

6. The computer-executed method of claim 5 , further comprising:
programmatically analyzing the first textual content to retrieve a third semantic concept,
wherein programmatically generating the suggested response to the one or more messages is further based on the third semantic concept.

7. The computer-executed method of claim 1 , further comprising:
determining that a fourth semantic concept is associated with the first semantic concept; and
determining second textual content based on an association of the second textual content with the fourth semantic concept, wherein the suggested response includes the second textual content.

8. The computer-executed method of claim 1 , wherein programmatically generating the suggested response further comprises:
determining a respective score for each of a plurality of suggested responses; and
selecting, based on the respective scores for the plurality of suggested responses, a particular response from the plurality of suggested responses as the suggested response.

9. The computer-executed method of claim 1 , further comprising filtering previous user responses to a plurality of previous images such that filtered previous user responses are statistically associated with the plurality of previous images,
wherein programmatically generating the suggested response is based on the filtered previous user responses to the plurality of previous images.

10. The computer-executed method of claim 1 , wherein retrieving the first semantic concept comprises querying a hierarchical taxonomy of concepts based on the first image content.

11. The computer-executed method of claim 1 , wherein programmatically generating the suggested response comprises generating the suggested response using at least one of a graph-based learning model or one or more rules-based grammars.

12. The computer-executed method of claim 1 , wherein the suggested response includes textual content, and further comprising:
determining that a particular semantic concept is associated with the first semantic concept based on a predetermined relationship between the first semantic concept and the particular semantic concept in a stored taxonomy; and
determining the textual content based on an association of the textual content with the particular semantic concept.

13. The computer-executed method of claim 1 , wherein generating the suggested response includes determining the suggested response using a stored graph that indicates relationships between a plurality of concept nodes representing concepts, a plurality of image nodes representing images, and a plurality of response nodes representing responses, wherein the first semantic concept is matched to a first concept node and wherein the suggested response is determined based on strengths of connections between the first concept node and at least one of the response nodes.

14. The computer-executed method of claim 1 , wherein generating the suggested response includes using a grammar-based model in which the first semantic concept is matched to one or more concepts in a stored taxonomy and one or more related concepts are determined in the stored taxonomy, and wherein the one or more related concepts are input to one or more stored grammars associated with the one or more related concepts in the grammar-based model to generate the suggested response.

15. A system to automatically suggest content in a messaging application comprising:
a memory storing instructions; and
at least one processor configured to access the instructions in the memory to perform operations comprising:
receiving one or more messages including a first image and a second image at a second device from a first device over a communication network;
posting, within the messaging application displayed on a second device, the one or more messages including the first image and the second image transmitted by the first device;
programmatically generating a suggested response to the one or more messages, the suggested response generated based on a first semantic concept associated with a first image content in the first image and a second semantic concept associated with a second image content in the second image; and
transmitting a command to cause rendering of the suggested response in the messaging application as a suggested response to the one or more messages.

16. The system of claim 15 wherein the at least one processor is configured to perform additional operations comprising:
receiving a selection of the suggested response in the messaging application based on user input provided via the second device; and
transmitting the selected suggested response to the first device.

17. The system of claim 15 , wherein the first semantic concept and the second semantic concept each include text, and wherein the suggested response includes the text from the first semantic concept and the text from the second semantic concept.

18. The system of claim 15 , wherein the suggested response includes a suggested image rendered in the messaging application, wherein the suggested image depicts a first feature related to the first semantic concept and a second feature related to the second semantic concept.

19. A non-transitory computer readable medium having stored thereon software instructions that, when executed by a processor, cause the processor to automatically suggest content in a messaging application by performing operations including:
receiving one or more messages including a first image and a second image a

Assignees

• Google Llc

Inventors

• Mcgregor Jr John Patrick
• Cassidy Ryan
• Fuxman Ariel
• Ramavajjala Vivek
• Ravi Sujith
• Nazarov Sergey
• Fulay Amit

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• H04L51/52

  for supporting social networking services · CPC title

• G06V30/274

  Syntactic or semantic context, e.g. balancing · CPC title

• H04L51/02

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• H04L51/10 Primary

  Multimedia information · CPC title