Multi-assistant natural language input processing

Patent metadata

Field	Value
Publication number	US-11636851-B2
Application number	US-202117387157-A
Country	US
Kind code	B2
Filing date	Jul 28, 2021
Priority date	Sep 24, 2019
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

Techniques for a natural language processing (NLP) system to implement more than one assistant are described. The NLP system may receive a natural language input corresponding to more than one user command. The NLP system may respond to a first command, of the natural language input, using a TTS voice of a first NLP system assistant. The NLP system may respond to a second command, of the natural language input, using a TTS voice of a second NLP system assistant.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving, by a system corresponding to at least a first assistant voice and a second assistant voice, first audio data representing a first spoken natural language input;
performing speech processing using the first audio data to determine a first portion corresponding to a first command;
determining first output data responsive to the first spoken natural language input;
based at least in part on the first command, selecting first data configured to generate synthesized speech in the first assistant voice;
performing speech synthesis processing using the first data and the first output data to generate second audio data; and
causing the second audio data to be output.

2. The computer-implemented method of claim 1 , wherein the first command corresponds to a first wakeword associated with the first assistant voice.

3. The computer-implemented method of claim 1 , wherein the first assistant voice corresponds to a first virtual assistant and wherein the method further comprises:
determining at least a portion of the first output data based at least in part on the first virtual assistant.

4. The computer-implemented method of claim 1 , further comprising:
receiving third audio data representing a second spoken natural language input corresponding to a same dialog as the first spoken natural language input;
performing speech processing using the third audio data to determine a second portion corresponding to a second command;
determining second output data responsive to the second spoken natural language input;
based at least in part on the second command, selecting second data configured to generate synthesized speech in the second assistant voice;
performing speech synthesis processing using the second data and the second output data to generate fourth audio data; and
causing the fourth audio data to be output.

5. The computer-implemented method of claim 1 , further comprising:
determining second output data responsive to a second portion of the first spoken natural language input;
based at least in part on the second portion, selecting second data configured to generate synthesized speech in the second assistant voice;
performing speech synthesis processing using the second data and the second output data to generate fourth audio data; and
causing the fourth audio data to be output.

6. The computer-implemented method of claim 5 , further comprising:
including, in the first output data, an identifier corresponding to the second assistant voice,
wherein the second audio data includes a representation of the identifier.

7. The computer-implemented method of claim 1 , further comprising:
processing the first audio data to determine a user identifier corresponding to the first spoken natural language input,
wherein selection of the first assistant voice is further based at least in part on the user identifier.

8. The computer-implemented method of claim 1 , further comprising:
determining use of the first assistant voice is enabled with respect to the first spoken natural language input.

9. The computer-implemented method of claim 1 , wherein the first audio data is received from a first device and the method further comprises:
after selecting the first data, associating an assistant identifier with the first device, wherein the assistant identifier corresponds to the first assistant voice.

10. The computer-implemented method of claim 1 , wherein the first audio data is received from a first device and the method further comprises:
after causing the second audio data to be output, deleting an association between an assistant identifier and the first device, wherein the assistant identifier corresponds to the first assistant voice.

11. A system comprising:
at least one processor; and
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
receive, by a system corresponding to at least a first assistant voice and a second assistant voice, first audio data representing a first spoken natural language input;
perform speech processing using the first audio data to determine a first portion corresponding to a first command;
determine first output data responsive to the first spoken natural language input;
based at least in part on the first command, select first data configured to generate synthesized speech in the first assistant voice;
perform speech synthesis processing using the first data and the first output data to generate second audio data; and
cause the second audio data to be output.

12. The system of claim 11 , wherein the first command corresponds to a first wakeword associated with the first assistant voice.

13. The system of claim 11 , wherein the first assistant voice corresponds to a first virtual assistant and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine at least a portion of the first output data based at least in part on the first virtual assistant.

14. The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
receive third audio data representing a second spoken natural language input corresponding to a same dialog as the first spoken natural language input;
perform speech processing using the third audio data to determine a second portion corresponding to a second command;
determine second output data responsive to the second spoken natural language input;
based at least in part on the second command, select second data configured to generate synthesized speech in the second assistant voice;
perform speech synthesis processing using the second data and the second output data to generate fourth audio data; and
cause the fourth audio data to be output.

15. The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine second output data responsive to a second portion of the first spoken natural language input;
based at least in part on the second portion, select second data configured to generate synthesized speech in the second assistant voice;
perform speech synthesis processing using the second data and the second output data to generate fourth audio data; and
cause the fourth audio data to be output.

16. The system of claim 15 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
include, in the first output data, an identifier corresponding to the second assistant voice,
wherein the second audio data includes a representation of the identifier.

17. The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
process the first audio data to determine a user identifier corresponding to the first spoken natural language input,
wherein selection of the first assistant voice is further based at least in part on the user identifier.

18. The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine use of the first assistant voice is enabled with respect to the first spoken natural language input.

Assignees

• Amazon Tech Inc

Inventors

• Mahmood Munir
• Bushkin Leopold
• Loeb Alexander Thomas
• Schwartz Michael
• Arif Mohammed
• Shen Rongzhou
• Gundeti Vikram Kumar
• Anwar Shemyla
• Khan Yaser
• Foyle Edward Page
• Li Bo

Classifications

• G10L13/033 Primary

  Voice editing, e.g. manipulating the voice of the synthesiser · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L13/00

  Speech synthesis; Text to speech systems · CPC title