Automatic response suggestions based on images received in messaging applications

Patent metadata

Field	Value
Publication number	US-10015124-B2
Application number	US-201715709418-A
Country	US
Kind code	B2
Filing date	Sep 19, 2017
Priority date	Sep 20, 2016
Publication date	Jul 3, 2018
Grant date	Jul 3, 2018

Abstract

Implementations relate to automatic response suggestions based on images received in messaging applications. In some implementations, a computer-executed method includes detecting a first image included within a first message received at a second device over a communication network from a first device of a first user, and programmatically analyzing the first image to extract a first image content. The method includes retrieving a first semantic concept associated with the first image content, programmatically generating a suggested response to the first message based on the first semantic concept, and transmitting instructions causing rendering of the suggested response in the messaging application as a suggestion to a second user of the second device.

First claim

What is claimed is:

1. A computer-executed method to automatically suggest content in a messaging application, the computer-executed method comprising:
detecting a first image included within a first message received at a second device over a communication network from a first device of a first user;
programmatically analyzing the first image to extract a first image content;
retrieving a first semantic concept associated with the first image content;
programmatically generating a suggested response to the first message based on the first semantic concept; and
transmitting instructions causing rendering of the suggested response in the messaging application as a suggestion to a second user of the second device.

2. The computer-executed method of claim 1 , further comprising:
upon receiving a selection of the suggested response based on input received from the second user, transmitting the suggested response over the communication network to a device of the first user as a response to the first message.

3. The computer-executed method of claim 1 , further comprising:
detecting a first textual content within the first message, wherein the suggested response is generated further based on the first textual content of the first message.

4. The computer-executed method of claim 3 , further comprising:
programmatically analyzing the first textual content to retrieve a second semantic concept, wherein the suggested response is generated further based on the second semantic concept.

5. The computer-executed method of claim 1 , wherein the suggested response includes a second image.

6. The computer-executed method of claim 5 , further comprising:
determining that a third semantic concept is associated with the first semantic concept;
programmatically retrieving a second image content associated with the third semantic concept; and
selecting the second image for the suggested response based on the second image having the second image content.

7. The computer-executed method of claim 1 , wherein the suggested response includes second textual content, and further comprising:
determining that a fourth semantic concept is associated with the first semantic concept; and
determining the second textual content based on an association of the second textual content with the fourth semantic concept.

8. The computer-executed method of claim 1 , wherein retrieving the first semantic concept comprises querying a hierarchical taxonomy of concepts based on the first image content.

9. The computer-executed method of claim 1 , wherein programmatically generating the suggested response comprises generating the suggested response using one or more of a graph-based learning model and a grammar-based model.

10. The computer-executed method of claim 1 , wherein programmatically generating the suggested response further comprises:
determining a respective score for one or more of a plurality of suggested responses that include the suggested response; and
selecting the suggested response based on the respective scores for the one or more of the plurality of suggested responses.

11. A non-transitory computer readable medium having stored thereon software instructions that, when executed by a processor, cause the processor to automatically suggest content in a messaging application by performing operations including:
posting, within the messaging application displayed on a second device, a first message including a first image transmitted by a first device of a first user;
programmatically generating one or more suggested responses to the first message, the one or more suggested responses generated based on a first semantic concept associated with a first image content in the first image;
transmitting instructions causing rendering of at least one suggested response of the one or more suggested responses in the messaging application as one or more suggestions to a second user; and
receiving a selection of a selected suggested response of the at least one suggested response in the messaging application based on user input provided by the second user of the second device.

12. The non-transitory computer readable medium of claim 11 , wherein the operation of programmatically generating the one or more suggested responses comprises generating the one or more suggested responses using one or more of a graph-based learning model and a grammar-based model.

13. The non-transitory computer readable medium of claim 11 , wherein the operation of programmatically generating the one or more suggested responses comprises programmatically generating a plurality of suggested responses, and wherein the operations further comprise:
determining a subset of the plurality of suggested responses, wherein transmitting instructions causing rendering of the at least one suggested response includes transmitting instructions causing rendering of the subset of the plurality of suggested responses.

14. The non-transitory computer readable medium of claim 11 , wherein the operation of programmatically generating the one or more suggested responses is based on data indicating a plurality of previous user responses to a plurality of previous images, wherein the previous user responses are filtered to be statistically associated with the plurality of previous images, wherein the statistically associated responses satisfy a threshold association score.

15. The non-transitory computer readable medium of claim 11 , wherein the at least one suggested response includes textual content, and the operations further comprise:
determining that a second semantic concept is associated with the first semantic concept based on a predetermined relationship between the first semantic concept and the second semantic concept in a stored taxonomy; and
determining the textual content based on an association of the textual content with the second semantic concept.

16. The non-transitory computer readable medium of claim 11 , wherein the operation of generating the at least one suggested response includes determining the one or more suggested responses using a stored graph providing relationships between a plurality of concept nodes representing concepts, a plurality of image nodes representing images, and a plurality of response nodes representing responses, wherein the first semantic concept is matched to a concept node and wherein the one or more suggested responses are determined based on strengths of connections between at least one of the concept nodes and at least one of the response nodes.

17. The non-transitory computer readable medium of claim 16 wherein:
multiple image nodes of the plurality of image nodes are connected in the stored graph based on a visual similarity of image pixels between images represented by the multiple image nodes; and
multiple response nodes of the plurality of response nodes are connected in the stored graph based on correlations between responses represented by the multiple response nodes, wherein the correlations include at least one of:
similarity of the responses represented by the multiple response nodes; or
frequency of occurrence of the responses represented by the multiple response nodes in response to images having particular concepts.

18. The non-transitory computer readable medium of claim 11 , wherein the operation of generating the at least one suggested response includes determining one or more of the suggested responses using a grammar-based model in which the first semantic concept is matched to one or more concepts in a stored taxonomy and one or more related

Assignees

• Google Llc

Inventors

• Mcgregor Jr John Patrick
• Cassidy Ryan
• Fuxman Ariel
• Ramavajjala Vivek
• Ravi Sujith
• Nazarov Sergey
• Fulay Amit

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• H04L51/10 Primary

  Multimedia information · CPC title

• H04L51/02

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• G06K9/6202

  Physics · mapped topic