Sentiment aware voice user interface

Patent metadata

Field	Value
Publication number	US-11508361-B2
Application number	US-202016889420-A
Country	US
Kind code	B2
Filing date	Jun 1, 2020
Priority date	Jun 1, 2020
Publication date	Nov 22, 2022
Grant date	Nov 22, 2022

Abstract

Described herein is a system for responding to a frustrated user with a response determined based on spoken language understanding (SLU) processing of a user input. The system detects user frustration and responds to a repeated user input by confirming an action to be performed or presenting an alternative action, instead of performing the action responsive to the user input. The system also detects poor audio quality of the captured user input, and responds by requesting the user to repeat the user input. The system processes sentiment data and signal quality data to respond to user inputs.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving first audio data representing a first utterance;
associating the first audio data with a first dialogue session identifier;
determining, using automatic speech recognition (ASR) processing, first ASR output data corresponding to the first audio data;
determining, using natural language understanding (NLU) processing, a first NLU hypothesis corresponding to the first ASR output data, the first NLU hypothesis associated with a first confidence score;
determining, using natural language understanding (NLU) processing, a second NLU hypothesis corresponding to the first ASR output data, the second NLU hypothesis associated with a second confidence score;
associating at least the first ASR output data, the first NLU hypothesis and the second NLU hypothesis with the first dialogue session identifier;
performing a first action in response to the first utterance, the first action corresponding to an intent included in the first NLU hypothesis;
receiving second audio data representing a second utterance;
associating the second audio data with the first dialogue session identifier;
determining second ASR output data corresponding to the second audio data;
determining a third NLU hypothesis corresponding to the second ASR output data;
determining a fourth NLU hypothesis corresponding to the second ASR output data;
associating at least the second ASR output data, the third NLU hypothesis and the fourth NLU hypothesis with the first dialogue session identifier;
determining, using the first dialogue session identifier, that the second utterance is a repeat of the first utterance based at least in part on a comparison of the first NLU hypothesis and the third NLU hypothesis;
receiving sentiment data indicating a sentiment based on acoustic characteristics of the second audio data;
determining that the sentiment data indicates frustration;
determining, using the first dialogue session identifier, that the second NLU hypothesis corresponds to the fourth NLU hypothesis;
determining that the second confidence score satisfies a threshold value;
in response to (i) determining that the sentiment data indicates frustration, (ii) determining that the second utterance repeats the first utterance, and (iii) that the second confidence score satisfies the threshold value: determining output text data including a representation of a second action corresponding to the second NLU hypothesis;
determining output audio data corresponding to the output text data using text-to-speech (TTS) processing; and
sending the output audio data to a device.

2. The computer-implemented method of claim 1 , further comprising:
receiving third audio data representing a third utterance;
associating the third audio data with a second dialogue session identifier;
determining, using ASR processing, third ASR output data corresponding to the third audio data;
determining, using NLU processing, a fifth NLU hypothesis and a sixth NLU hypothesis corresponding to the third ASR output data, the fifth NLU hypothesis associated with a third confidence score and the sixth NLU hypothesis associated with a fourth confidence score;
performing a third action corresponding to the fifth NLU hypothesis;
receiving fourth audio data representing a fourth utterance;
associating the fourth audio data with the second dialogue session identifier;
determining, using ASR, fourth ASR output data corresponding to the fourth audio data;
determining, using the second dialogue session identifier, that the fourth utterance is a repeat of the third utterance;
receiving second sentiment data corresponding to the fourth audio data, the second sentiment data indicating a sentiment based on acoustic characteristics of the fourth audio data;
determining that the second sentiment data indicates frustration;
determining, using the second dialogue session identifier, that the sixth NLU hypothesis corresponds to the fourth ASR output data;
determining that the fourth confidence score does not satisfy the threshold value;
in response to determining that the second sentiment data indicates frustration, determining second output text data including a confirmation to perform the third action corresponding to the fourth NLU hypothesis;
determining second output audio data corresponding to the second output text data using TTS processing; and
sending the second output audio data to the device.

3. The computer-implemented method of claim 1 , further comprising:
receiving third audio data representing a third utterance;
associating the third audio data with a second dialogue session identifier;
determining, using ASR processing, third ASR output data corresponding to the third audio data;
determining, using NLU processing, a fifth NLU hypothesis corresponding to the third ASR output data, a sixth NLU hypothesis associated with a third confidence score;
receiving signal-to-noise ratio (SNR) data corresponding to the third audio data;
determining that the SNR data exceeds a second threshold value indicating signal energy associated with the third utterance is low;
in response to determining that the SNR data exceeds the second threshold value, determining second output text data including a request to move closer to the device and repeat the third utterance;
determining second output audio data corresponding to the second output text data using TTS processing; and
sending the second output audio data to the device.

4. The computer-implemented method of claim 1 , further comprising:
receiving third audio data representing a third utterance;
associating the third audio data with the first dialogue session identifier;
determining, using ASR processing, third ASR output data corresponding to the third audio data;
determining, using NLU processing, that the third ASR output data corresponds to negative feedback;
determining second output text data representing an apology;
determining second output audio data corresponding to the second output text data using TTS processing;
sending the second output audio data to the device;
determining to end a dialogue corresponding to the first dialogue session identifier; and
associating subsequently received fourth audio data with a second dialogue session identifier.

5. A computer-implemented method comprising:
receiving first audio data representing a first utterance;
determining, using natural language understanding (NLU) processing, first NLU data corresponding to the first audio data;
determining, using NLU processing, second NLU data corresponding to the first audio data, the second NLU data different than the first NLU data;
causing a first action to be performed in response to the first utterance, the first action corresponding to an intent included in the first NLU data;
receiving second audio data representing a second utterance;
determining a repeat indicator based on the second utterance being semantically similar to the first utterance;
receiving, from a sentiment detection component, sentiment data corresponding to the second audio data;
determining that the sentiment data indicates frustration; and
in response to the repeat indicator and the sentiment data indicating frustration, determining output data including a representation of a second action corresponding to the second NLU data, wherein the second action is different from the first action.

6. The computer-implemented method of claim 5 , further comprising:
determining, using NLU processing, second NLU data corresponding to the first audio data, the second NLU data different than the first NLU data;
determining that the first NLU data satisfies a first condition;
determining that the second NLU data does not satisfy a seco

Assignees

• Amazon Tech Inc

Inventors

• Madwed Isaac Joseph
• Nemer Julia Kennedy
• Kim Joo-Kyung
• Strom Nikko
• Saunders Steven Mack
• Panfili Laura Maggia
• Jentoft Anna Caitlin
• Lee Sungjin
• Thomas David
• Kim Young-Bum
• Ganga Pablo Cesar
• Guo Chenlei
• Tang Shuting
• Yao Zhenyu

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/1815 Primary

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G06F2218/12

  Classification; Matching · CPC title

• G10L2015/225

  Feedback of the input speech · CPC title