Processing complex utterances for natural language understanding

Patent metadata

Field	Value
Publication number	US-11410646-B1
Application number	US-201916368399-A
Country	US
Kind code	B1
Filing date	Mar 28, 2019
Priority date	Sep 29, 2016
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

A system capable of performing natural language understanding (NLU) on utterances including complex command structures such as sequential commands (e.g., multiple commands in a single utterance), conditional commands (e.g., commands that are only executed if a condition is satisfied), and/or repetitive commands (e.g., commands that are executed until a condition is satisfied). Audio data may be processed using automatic speech recognition (ASR) techniques to obtain text. The text may then be processed using machine learning models that are trained to parse text of incoming utterances. The models may identify complex utterance structures and may identify what command portions of an utterance go with what conditional statements. Machine learning models may also identify what data is needed to determine when the conditionals are true so the system may cause the commands to be executed (and stopped) at the appropriate times.

First claim

What is claimed is:

1. A system, comprising:
at least one processor; and
at least one computer-readable medium encoded with instructions that, when executed by the at least one processor, cause the system to:
receive audio data corresponding to a first utterance from a first device;
process the audio data to determine that a first portion of the audio data corresponds to a command and that a second portion of the audio data corresponds to a conditional statement;
determine that the conditional statement corresponds to a first data type;
receive, from a second device, first data having the first data type;
cause, at a first time and based at least in part on the command, at least one component to transition from a first state to a second state;
determine, using the first data and at a second time after the first time, that the conditional statement is satisfied; and
cause the at least one component to transition from the second state to the first state based at least in part on the conditional statement being satisfied.

2. The system of claim 1 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine a user profile associated with the first device; and
determine that the second device is associated with the user profile.

3. The system of claim 2 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine that the first data type corresponds to a sensor type; and
determine that the second device corresponds to the sensor type.

4. The system of claim 1 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine text data associated with the first portion; and
associate the text data with a first indicator corresponding to the second device and a second indicator corresponding to a value corresponding to the first data type.

5. The system of claim 1 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
receive the first data during a same communication session as the audio data.

6. The system of claim 1 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine that the first data type corresponds to information available from an internet data source; and
determine that the second device corresponds to the internet data source.

7. The system of claim 1 , wherein the audio data further includes a third portion and the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine that the conditional statement corresponds to the command and not the third portion.

8. The system of claim 7 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine that the third portion corresponds to a second command; and
cause the second command to be executed regardless of the conditional statement.

9. A system, comprising:
at least one processor; and
at least one computer-readable medium encoded with instructions that, when executed by the at least one processor, cause the system to:
receive audio data from a first device;
process the audio data to determine that a first portion of the audio data corresponds to a request to repeat execution of a command and that a second portion of the audio data corresponds to a conditional statement;
cause the command to be executed multiple times, beginning at a first time;
determine, at a second time after the first time, that the conditional statement is satisfied; and
ceasing from further executing the command based at least in part on the conditional statement being satisfied.

10. The system of claim 9 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine that the conditional statement corresponds to a number of times to repeat execution of the command, wherein the number of times is greater than one;
determine that the conditional statement is satisfied by determining that the command has been executed the number of times; and
ceasing from further executing the command based at least in part on the command having been executed the number of times.

11. The system of claim 9 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine that the conditional statement corresponds to a first time;
determine that the conditional statement is satisfied at least in part by determining that a current time corresponds to the first time; and
ceasing from further executing the command based at least in part on the current time corresponding to the first time.

12. The system of claim 9 , wherein the first time is a time of day.

13. The system of claim 9 , wherein the audio data further includes a third portion and the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine that the conditional statement corresponds to the command and not the third portion.

14. The system of claim 13 , wherein the at least one computer-readable medium is encoded with additional instructions that, when executed by the at least one processor, further cause the system to:
determine that the third portion corresponds to a second command; and
cause the second command to be executed regardless of the conditional statement.

15. A computer-implemented method, comprising:
receiving audio data corresponding to a first utterance from a first device;
processing the audio data to determine that a first portion of the audio data corresponds to a command and that a second portion of the audio data corresponds to a conditional statement;
determining that the conditional statement corresponds to a first data type;
receiving, from a second device, first data having the first data type;
causing, at a first time and based at least in part on the command, at least one component to transition from a first state to a second state;
determining, using the first data and at a second time after the first time, that the conditional statement is satisfied; and
causing the at least one component to transition from the second state to the first state based at least in part on the conditional statement being satisfied.

16. The computer-implemented method of claim 15 , further comprising:
determining a user profile associated with the first device; and
determining that the second device is associated with the user profile.

17. The computer-implemented method of claim 16 , further comprising:
determining that the first data type corresponds to a sensor type; and
determining that the second device corresponds to the sensor type.

18. The computer-implemented method of claim 15 , further comprising:
determining text data associated with the fir

Assignees

• Amazon Tech Inc

Inventors

• Erbas Cengiz
• Kollar Thomas
• Sikka Avnish
• Matsoukas Spyridon
• Reavely Simon Peter

Classifications

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/063

  Training · CPC title