Memory grounded conversational reasoning and question answering for assistant systems

Patent metadata

Field	Value
Publication number	US-11657094-B2
Application number	US-201916552559-A
Country	US
Kind code	B2
Filing date	Aug 27, 2019
Priority date	Jun 28, 2019
Publication date	May 23, 2023
Grant date	May 23, 2023

Abstract

In one embodiment, a method includes receiving a query from a user from a client system associated with the user, determining one or more initial memory slots based on the query, accessing a memory graph associated with the user which comprises a plurality of nodes and a plurality of edges connecting the nodes, and wherein one or more of the nodes correspond to one or more episodic memories of the user, respectively, and wherein each edge corresponds to a relationship between the connected nodes, selecting one or more candidate nodes from the memory graph by one or more machine-learning models based on the initial memory slots, generating a response based on the initial memory slots and episodic memories corresponding to the selected candidate nodes, and sending instructions for presenting the response to the client system in response to the query.

First claim

What is claimed is:

1. A method comprising, by an assistant system:
receiving, from a client system associated with a user via an assistant xbot associated with the assistant system, a query from the user;
determining, based on the query, one or more initial memory slots;
accessing a memory graph associated with the user, wherein the memory graph comprises a plurality of nodes and a plurality of edges connecting the nodes, and wherein one or more of the nodes correspond to one or more episodic memories of the user, respectively, and wherein each edge corresponds to a relationship between the connected nodes;
determining, by one or more machine-learning models based on the initial memory slots and dialog contexts associated with the query, one or more sequence paths of walk steps traversing the memory graph;
selecting, by the one or more machine-learning models based on the one or more sequence paths of walk steps, one or more candidate nodes from the memory graph;
generating a response based on the initial memory slots and episodic memories corresponding to the selected candidate nodes; and
sending, to the client system in response to the query via the assistant xbot, instructions for presenting the response.

2. The method of claim 1 , wherein the one or more candidate nodes are selected from the one or more nodes corresponding to the one or more episodic memories of the user.

3. The method of claim 1 , wherein one or more nodes of the plurality of nodes correspond to one or more entities, respectively.

4. The method of claim 1 , wherein the memory graph is built based on a knowledge graph comprising a plurality of nodes corresponding to a plurality of entities, respectively.

5. The method of claim 4 , wherein the plurality of nodes of the memory graph comprise:
a plurality of nodes corresponding to a plurality of entities from the knowledge graph; and
a plurality of nodes corresponding to a plurality of episodic memories of the user.

6. The method of claim 4 , further comprising:
building a synthetic memory graph generator; and
generating, by the synthetic memory graph generator, one or more nodes in the memory graph, wherein each of the one or more nodes is connected to one or more entities associated with the knowledge graph.

7. The method of claim 1 , wherein the query is associated with a context, and wherein selecting the one or more candidate nodes is further based on the context associated with the query.

8. The method of claim 1 , wherein the one or more machine-learning models comprise at least a long-short term memory (LSTM) model.

9. The method of claim 1 , further comprising:
determining relevance between the initial memory slots and each of the plurality of nodes in the memory graph, wherein selecting the one or more candidate nodes is further based on the relevance.

10. The method of claim 1 , further comprising:
generating a query encoding for the query based on a long-short term memory (LSTM) model; and
generating a memory encoding for each node corresponding to an episodic memory in the memory graph based on one or more of a graph embeddings projection model or a LSTM model.

11. The method of claim 10 , wherein selecting the one or more candidate nodes comprises inputting the generated query encoding and each generated memory encoding to the one or more machine-learning models, and wherein the one or more machine-learning models generate the one or more sequence paths of walk steps attending to the one or more candidate nodes within the memory graph.

12. The method of claim 11 , further comprising:
inputting the generated query encoding, each generated memory encoding, and the one or more sequence paths of walk steps to the one or more machine-learning models, wherein the one or more machine-learning models generate one or more answer candidates, wherein the answer candidates are ranked in an order based a compatibility between the query and the respective answer candidate, and wherein the compatibility is determined by the one or more machine-learning models.

13. The method of claim 12 , further comprising:
aggregating, by the one or more machine-learning models, the one or more answer candidates, wherein each answer candidate is associated with a weight, and wherein the weight is determined based on the rank associated with the respective answer candidate.

14. The method of claim 1 , wherein generating the response is further based on one or more language templates.

15. The method of claim 1 , wherein the response comprises one or more of a content file associated with the initial memory slots and episodic memories corresponding to the selected candidate nodes.

16. The method of claim 15 , wherein the content file comprises one or more of textual content, an image, a video clip, or an audio clip.

17. The method of claim 15 , wherein the response is presented in a user interface comprising one or more of a media section or a chat section, wherein the media section surfaces the image or video clip, and wherein the chat section surfaces the textual content.

18. The method of claim 1 , wherein the episodic memories of the user are extracted from one or more of:
a plurality of content objects associated with the user;
social-networking information associated with the user;
contact information associated with the user;
calendar information associated with the user; or
interactions with one or more other users in an online social network.

19. One or more computer-readable non-transitory storage media embodying software that is operable when executed by an assistant system to:
receive, from a client system associated with a user via an assistant xbot associated with the assistant system, a query from the user;
determine, based on the query, one or more initial memory slots;
access a memory graph associated with the user, wherein the memory graph comprises a plurality of nodes and a plurality of edges connecting the nodes, and wherein one or more of the nodes correspond to one or more episodic memories of the user, respectively, and wherein each edge corresponds to a relationship between the connected nodes;
determine, by one or more machine-learning models based on the initial memory slots and dialog contexts associated with the query, one or more sequence paths of walk steps traversing the memory graph;
select, by the one or more machine-learning models based on the one or more sequence paths of walk steps, one or more candidate nodes from the memory graph;
generate a response based on the initial memory slots and episodic memories corresponding to the selected candidate nodes; and
send, to the client system in response to the query via the assistant xbot, instructions for presenting the response.

20. An assistant system comprising: one or more processors;
and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, from a client system associated with a user via an assistant xbot associated with the assistant system, a query from the user;
determine, based on the query, one or more initial memory slots;
access a memory graph associated with the user, wherein the memory graph comprises a plurality of nodes and a plurality of edges connecting the nodes, and wherein one or more of the nodes correspond to one or more episodic memories of the user, respectively, and wherein each edge corresponds to a relationship between the connected nodes;
determin

Assignees

• Meta Platforms Tech Llc

Inventors

• Moon Seungwhan
• Shah Pararth Paresh
• Kumar Anuj
• Subba Rajen

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06F16/9035

  Filtering based on additional data, e.g. user or group profiles · CPC title

• G06N3/045

  Combinations of networks · CPC title