End-to-end spoken language understanding without full transcripts

Patent metadata

Field	Value
Publication number	US-11929062-B2
Application number	US-202017021956-A
Country	US
Kind code	B2
Filing date	Sep 15, 2020
Priority date	Sep 15, 2020
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

A method and system of training a spoken language understanding (SLU) model includes receiving natural language training data comprising (i) one or more speech recording, and (ii) a set of semantic entities and/or intents for each corresponding speech recording. For each speech recording, one or more entity labels and corresponding values, and one or more intent labels are extracted from the corresponding semantic entities and/or overall intent. A spoken language understanding (SLU) model is trained based upon the one or more entity labels and corresponding values, and one or more intent labels of the corresponding speech recordings without a need for a transcript of the corresponding speech recording.

First claim

What is claimed is:

1. A computing device comprising:
a processor;
a network interface coupled to the processor to enable communication over a network;
an engine configured to perform acts comprising, during a training phase of a spoken language understanding (SLU) model:
receiving, over the network, natural language training data comprising (i) one or more speech recording, and (ii) a set of semantic entities and/or an overall intent for each corresponding speech recording;
for each speech recording, extracting (i) one or more entity labels and corresponding values, and (ii) one or more intent labels from the corresponding semantic entities and/or overall intent; and
training the SLU model based upon, the one or more entity labels and corresponding values, and one or more intent labels of the corresponding speech recordings, without a need for a transcript of the corresponding speech recording.

2. The computing device of claim 1 , wherein the semantic entities are not in spoken order.

3. The computing device of claim 2 , wherein the semantic entities are in alphabetical order.

4. The computing device of claim 2 , wherein the natural language training data is based on a combination of different types of training data.

5. The computing device of claim 2 , wherein the engine is further configured to perform acts, comprising performing a pre-processing alignment to align the semantic entities into spoken order.

6. The computing device of claim 1 , wherein the training data is based on transaction data between a user and an administrator helping the user with a task.

7. The computing device of claim 1 , wherein the training data comprises a record of transaction data comprising a bag of entities.

8. The computing device of claim 1 , wherein the extraction of the one or more entity labels and corresponding values, and the one or more intent labels is by way of a neural network processing comprising at least one of connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), or attention-based encoder-decoder neural network.

9. The computing device of claim 1 , wherein the training involves a transfer learning comprising initializing the SLU model with an automatic speech recognition (ASR) model.

10. The computing device of claim 1 , wherein the SLU engine is further configured to perform acts, comprising, during an active phase:
receiving raw spoken language data comprising an audio speech recording without a transcript of the audio speech recording; and
using the trained SLU model to recognize a meaning of the raw spoken language data, wherein the meaning comprises an intent and semantic entities of the raw spoken language.

11. A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out a method of training a spoken language understanding (SLU) model, comprising:
receiving natural language training data comprising (i) one or more speech recording, and (ii) a set of semantic entities and/or an overall intent for each corresponding speech recording;
for each speech recording, extracting (i) one or more entity labels and corresponding values, and (ii) one or more intent labels from the corresponding semantic entities and/or overall intent; and
training the spoken language understanding (SLU) model based upon the one or more entity labels and corresponding values, and one or more intent labels of the corresponding speech recordings, without a need for a transcript of the corresponding speech recording.

12. The non-transitory computer readable storage medium of claim 11 , wherein the semantic entities are not in spoken order.

13. The non-transitory computer readable storage medium of claim 12 , wherein the natural language training data is based on a combination of different types of training data.

14. The non-transitory computer readable storage medium of claim 13 , the method further comprising performing a pre-processing alignment to align the semantic entities into spoken order.

15. The non-transitory computer readable storage medium of claim 11 , wherein the training data is based on transaction data between a user and an administrator helping the user with a task.

16. The non-transitory computer readable storage medium of claim 11 , wherein the training data comprises a record of transaction data comprising a bag of entities.

17. The non-transitory computer readable storage medium of claim 11 , wherein the extraction of the one or more entity labels and corresponding values, and the one or more intent labels is by way of a neural network processing comprising at least one of connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), or attention-based encoder-decoder neural network.

18. The non-transitory computer readable storage medium of claim 11 , wherein the training involves a transfer learning comprising initializing the SLU model with an automatic speech recognition (ASR) model.

19. The non-transitory computer readable storage medium of claim 11 , the method further comprising, during an active phase:
receiving raw spoken language data comprising an audio speech recording without a transcript of the audio speech recording; and
using the trained SLU model to recognize a meaning of the raw spoken language data, wherein the meaning comprises an intent and semantic entities of the raw spoken language.

20. A computer implemented method comprising:
during a training phase of a spoken language understanding (SLU) model, receiving natural language training data comprising (i) one or more speech recording, and (ii) a set of semantic entities and/or an overall intent for each corresponding speech recording;
for each speech recording, extracting (i) one or more entity labels and corresponding values, and (ii) one or more intent labels from the corresponding semantic entities and/or overall intent; and
training the SLU model based upon the one or more entity labels and corresponding values, and one or more intent labels of the corresponding speech recordings, without a need for a transcript of the corresponding speech recording.

Assignees

• IBM

Inventors

• Kuo Hong-Kwang Jeff
• Tueske Zoltan
• Thomas Samuel
• Huang Yinghui
• Kingsbury Brian E D
• Audhkhasi Kartik

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G10L15/16 Primary

  using artificial neural networks · CPC title