Generating multi-perspective responses by assistant systems

Patent metadata

Field	Value
Publication number	US-12374097-B2
Application number	US-202318164480-A
Country	US
Kind code	B2
Filing date	Feb 3, 2023
Priority date	Apr 20, 2018
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

In one embodiment, a method includes receiving a user query inputted on a head-mounted device from the head-mounted device, wherein the user query corresponds to multiple dialog-intents, executing multiple tasks corresponding to the multiple dialog-intents, generating a multi-perspective response by a stitching model based on two or more of execution results of the multiple tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results, and sending instructions to the head-mounted device for presenting the multi-perspective response on the head-mounted device.

First claim

What is claimed is:

1. A method comprising, by one or more computing systems:
receiving, from a computing device, a user query inputted on the computing device, wherein the user query corresponds to a plurality of dialog-intents;
executing a plurality of tasks corresponding to the plurality of dialog-intents;
generating, by a stitching model, a multi-perspective response based on two or more of execution results of the plurality of tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results; and
sending, to the computing device, instructions for presenting the multi-perspective response on the computing device.

2. The method of claim 1 , further comprising:
determining, based on the user query by a natural-language understanding module, the plurality of dialog-intents.

3. The method of claim 1 , wherein each of the plurality of dialog-intents is associated with a particular agent of a plurality of agents.

4. The method of claim 3 , further comprising:
receiving, from the plurality of agents, a plurality of execution results corresponding to the plurality of tasks, respectively; and
selecting the two or more of the execution results from the plurality of execution results for combination.

5. The method of claim 4 , further comprising:
determining respective relevance scores of the execution results with respect to the user query; and
ranking the execution results based on their respective relevance scores,
wherein selecting the two or more of the execution results for combination comprises selecting execution results based on their respective rankings.

6. The method of claim 4 , wherein selecting the two or more of the execution results for combination comprises:
determining, based on one or more machine-learning models, whether one or more of the execution results are mutually exclusive results with respect to one or more of the other execution results; and
filtering one or more of the mutually exclusive results from the execution results,
wherein selecting the two or more of the execution results for combination comprises selecting two or more execution results from the post-filtered execution results.

7. The method of claim 4 , wherein selecting the two or more of the execution results for combination comprises:
calculating, for each execution result of the execution results, an entropy value based on an information entropy of the respective execution result; and
calculating, for each execution result of the execution results, information-gain values of the execution result based on its respective entropy value with respect to the entropy values of each other execution result of the execution results,
wherein selecting the two or more of the execution results for combination comprises selecting two or more execution results based on their respective information-gain values.

8. The method of claim 1 , further comprising:
determining, by a natural-language understanding module, one or more slots associated with each of the plurality of dialog-intents.

9. The method of claim 1 , wherein generating the multi-perspective response comprises:
determining an order of the two or more of the execution results; and
combining, by the stitching model, the two or more of the execution results based on the determined order.

10. The method of claim 9 , wherein the order of the two or more of the execution results is determined based on a sequential-language model.

11. The method of claim 10 , wherein the sequential-language model is trained based on a plurality of training data of human-combined utterances.

12. The method of claim 9 , wherein the order of the two or more of the execution results is determined based on one or more predefined rules.

13. The method of claim 9 , wherein the user query is associated with a first user, and wherein the order of the two or more of the execution results is determined based on user profile data associated with the first user.

14. The method of claim 9 , wherein the order of the two or more of the execution results is determined based on execution times associated with the respective execution results.

15. The method of claim 9 , wherein the user query is associated with a first user, and wherein the order of the two or more of the execution results is determined based on history data of user interactions by the first user with agents corresponding to the two or more of the execution results.

16. The method of claim 9 , wherein the order of the two or more of the execution results is determined based on linguistic grounding.

17. The method of claim 1 , wherein the multi-perspective response is based on one or more modalities.

18. The method of claim 17 , wherein the one or more modalities comprise one or more of text, audio, image, or video.

19. One or more computer-readable non-transitory non-volatile storage media embodying software that is operable when executed to:
receive, from a computing device, a user query inputted on the computing device, wherein the user query corresponds to a plurality of dialog-intents;
execute a plurality of tasks corresponding to the plurality of dialog-intents;
generate, by a stitching model, a multi-perspective response based on two or more of execution results of the plurality of tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results; and
send, to the computing device, instructions for presenting the multi-perspective response on the computing device.

20. A system comprising: one or more processors; and a non-transitory non-volatile memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, from a computing device, a user query inputted on the computing device, wherein the user query corresponds to a plurality of dialog-intents;
execute a plurality of tasks corresponding to the plurality of dialog-intents;
generate, by a stitching model, a multi-perspective response based on two or more of execution results of the plurality of tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results; and
send, to the computing device, instructions for presenting the multi-perspective response on the computing device.

Assignees

• Meta Platforms Inc

Inventors

• Koukoumidis Emmanouil
• Hanson Michael Robert
• Subba Rajen
• Young Heidi
• Shah Rushin
• Yu Jinsong
• Dumoulin Benoit F
• Kahn Jeremy Gillmor
• Iyer Chandrasekhar

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F21/6245

  Protecting personal data, e.g. for financial or medical purposes · CPC title