Audio message extraction

Patent metadata

Field	Value
Publication number	US-10319375-B2
Application number	US-201615392291-A
Country	US
Kind code	B2
Filing date	Dec 28, 2016
Priority date	Dec 28, 2016
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

Audio data, corresponding to an utterance spoken by a person within a detection range of a voice communications device, can include an audio message portion. The audio data can be captured and analyzed to determine the intent to send a message. Based at least in part upon that intent, a remaining portion of the audio data can be analyzed to determine the intended message target or recipient, as well as the portion corresponding to the actual message payload. Once determined, the audio file can be trimmed to the message payload, and the message payload of the audio data can be delivered as an audio message to the target recipient.

First claim

What is claimed is:

1. A system, comprising:
at least one device processor;
memory including instructions that, when executed by the at least one device processor, cause the system to:
receive audio input data from a voice communications device associated with an account, the audio input data corresponding to an utterance received by a microphone of the voice communications device, wherein a beginning of the utterance is identified by the voice communications device in response to a wakeword being detected by the voice communications device;
generate text data from the audio input data by performing automated speech recognition (ASR) on the audio input data;
determine, from the text data, a messaging intent by performing natural language processing (NLP) on the text data;
determine a slot pattern corresponding to the messaging intent, the slot pattern including at least a target slot and a message payload slot;
determine respective portions of the text data that correspond to the target slot and the message payload slot;
identify, based upon the text data corresponding to the target recipient slot and a contact list associated with the voice communications device, a recipient identifier;
determine a first timestamp associated with the message payload slot;
generate, based upon the first timestamp, audio message data including a portion of the audio data corresponding to the text data of the message payload slot; and
send the audio message data for playback on an audio playback device associated with the recipient identifier.

2. The system of claim 1 , wherein the instructions, when executed further cause the system to:
determine a second timestamp corresponding to an end location of the message payload slot; and
include only a portion of the audio input data located between a time of the first timestamp and a time of the second timestamp in the audio message data.

3. The system of claim 1 , wherein the instructions, when executed further cause the system to:
determine message text corresponding to the audio message data; and
transmit the message text for access via the recipient identifier, wherein the message text is able to be presented with, or separate from, the audio message data.

4. The system of claim 1 , wherein the instructions, when executed further cause the system to:
receive a request from the audio playback device to an address corresponding to the audio message data; and
send the audio message data for playback on the audio playback device in response to the request.

5. A computer-implemented method, comprising:
receiving audio input data corresponding to an utterance received by at least one microphone of a voice communications device associated with an account, wherein a beginning of the utterance is identified by the voice communications device in response to a wakeword being detected by the voice communications device;
determining a messaging intent represented by the audio input data;
determining a slot pattern corresponding to the messaging intent, the slot pattern including at least a target slot and a message payload slot;
determining, from the target slot, a recipient identifier represented by the audio input data;
determining a message payload portion that corresponds to the message payload slot;
determining a first time stamp identifying a beginning of the message payload portion; and
generating, for playback on an audio playback device and accessible according to the recipient identifier, audio message data including the message payload portion starting from a location of the first time stamp in the audio input data.

6. The computer-implemented method of claim 5 , further comprising:
determining a set of time stamps identifying locations of the target slot and the message payload slot, the set of time stamps including the first time stamp; and
determining the recipient identifier and the message payload portion based upon the locations of the set of time stamps with respect to the audio input data.

7. The computer-implemented method of claim 5 , further comprising:
generating tokenized text data from the audio input data by performing automated speech recognition (ASR) on the audio input data.

8. The computer-implemented method of claim 7 , further comprising:
determining at least the messaging intent, and respective words corresponding to the target slot and the message payload slot, by performing natural language processing (NLP) on the tokenized text data.

9. The computer-implemented method of claim 5 , further comprising:
determining, for the audio input data, an identity of a user having spoken the utterance; and
determining a contact list for the user based upon the identity; and
determining the recipient identifier based upon performing a lookup of a target from the target slot against the contact list for with the user.

10. The computer-implemented method of claim 5 , further comprising:
receiving media input data including the audio input data and corresponding video input data; and
extracting the audio input data for determining the messaging intent.

11. The computer-implemented method of claim 10 , further comprising:
including, for playback, a portion of the video input data corresponding to the message payload portion.

12. The computer-implemented method of claim 5 , further comprising:
determining that the recipient identifier is unable to be determined with at least a minimum level of confidence based on the audio input data;
causing additional audio input data to be received that includes additional identifying information for a target of the messaging intent; and
determining the recipient identifier based upon the additional identifying information.

13. The computer-implemented method of claim 5 , further comprising:
determining message text corresponding to the message payload slot; and
providing the message text for access via the recipient identifier, wherein the message text is able to be presented with, or separate from, the audio message data.

14. A system, comprising:
at least one device processor;
memory including instructions that, when executed by the at least one device processor, cause the system to:
receive media input data corresponding to an utterance received by a communications device associated with an account, wherein a beginning of the utterance is identified by the communications device in response to a wakeword being detected by the communications device;
extract audio input data from the media input data;
determine a messaging intent represented by the audio input data;
determine a slot pattern corresponding to the messaging intent, the slot pattern including at least a target slot and a message payload slot;
determine a recipient identifier represented by the audio input data;
determine a message payload portion that corresponds to the message payload slot;
determine a first time stamp identifying a beginning of the message payload portion; and
generate, for playback on a playback device and accessible according to the recipient identifier, media message data including the message payload portion starting from a location of the first time stamp in the audio input data.

15. The system of claim 14 , wherein the instructions, when executed further cause the system to:
determine a set of time stamps identifying locations of the target slot and the message payload slot in the audio input data; and
determine the recipient identifier and the message payload portion based upon the locations of the set of time stamps with respect to the audio input data.

Assignees

• Amazon Tech Inc

Inventors

• Fritz Neil Christopher
• Bhagat Lakshya
• Southwood Scott
• Doran Katelyn
• Lounsbury Brett
• Devaraj Christo Frank

Classifications

• G06F40/295

  Named entity recognition · CPC title

• G10L15/30

  Distributed recognition, e.g. in client-server systems, for mobile phones or network applications · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G10L2015/088

  Word spotting · CPC title