Data-driven and rule-based speech recognition output enhancement

Patent metadata

Field	Value
Publication number	US-11257484-B2
Application number	US-201916546715-A
Country	US
Kind code	B2
Filing date	Aug 21, 2019
Priority date	Aug 21, 2019
Publication date	Feb 22, 2022
Grant date	Feb 22, 2022

Abstract

According to some embodiments, a multi-layer speech recognition transcript post processing system may include a data-driven, statistical layer associated with a trained automatic speech recognition model that selects an initial transcript. A rule-based layer may receive the initial transcript from the data-driven, statistical layer and execute at least one pre-determined rule to generate a first modified transcript. A machine learning approach layer may receive the first modified transcript from the rule-based layer and perform a neural model inference to create a second modified transcript. A human editor layer may receive the second modified transcript from the machine learning approach layer along with an adjustment from at least one human editor. The adjustment may create, in some embodiments, a final transcript that may be used to fine-tune the data-driven, statistical layer.

First claim

What is claimed is:

1. A multi-layer speech recognition transcript post processing system, comprising:
a data-driven, statistical layer associated with a trained automatic speech recognition model that selects an initial transcript;
a rule-based layer that receives the initial transcript from the data-driven, statistical layer and executes at least one pre-determined rule to generate a first modified transcript;
a machine learning approach layer that receives the first modified transcript from the rule-based layer and performs a neural model inference to create a second modified transcript; and
a human editor layer that receives the second modified transcript from the machine learning approach layer and uses an adjustment received from at least one human editor to output a final transcript, wherein the final transcript is used to fine-tune the data-driven, statistical layer.

2. The system of claim 1 , wherein the data-driven, statistical layer selects a best initial transcript from a set of N most probable speech recognition transcripts.

3. The system of claim 2 , wherein the selection of the best initial transcript is augmented by external attention comprising multiple text documents.

4. The system of claim 1 , wherein the pre-determined rule is associated with at least one of: (i) a white list, (ii) a black list, and (iii) a rule approach.

5. The system of claim 4 , wherein the pre-determined rule is automatically generated via offline data mining, data augmentation, and model training.

6. The system of claim 5 , wherein the offline data mining is associated with at least one of: (i) supervised classification, (ii) unsupervised classification, (iii) clustering techniques, (iv) n-gram classification, (v) replacement pairs based on context, (vi) a graph-based method to link spoken and written sentences based on semantic similarity, and (vii) search engine data.

7. The system of claim 1 , wherein the machine learning approach layer is associated with at least one of: (i) online candidate generation, (ii) online neural model inference encoding and decoding, and (iii) online ranking.

8. The system of claim 1 , wherein the human editor layer is associated with at least one of: (i) multiple-level human labeling, (ii) pairwise human labeling, and (iii) manual human transcription.

9. The system of claim 8 , wherein the adjustment is associated with at least one of: (i) syntactic correctness, (ii) semantic closeness, (iii) fluency, and (iv) style.

10. The system of claim 1 , wherein the human editor layer includes a text-to-speech conversion followed by a speech-to-text conversion.

11. The system of claim 1 , wherein the final transcript is transmitted to a downstream task associated with at least one of: (i) language understanding, (ii) machine translation, (iii) text summarization, (iv) text classification, (v) information extraction, and (vi) question answering.

12. A computer-implemented method for a multi-layer speech recognition transcript post processing system, comprising:
selecting, by a data-driven, statistical layer associated with a trained automatic speech recognition model, an initial transcript;
receiving, by a rule-based layer, the initial transcript and executing at least one pre-determined rule to generate a first modified transcript;
receiving, by a machine learning approach layer, the first modified transcript from the rule-based layer and performing a neural model inference to create a second modified transcript; and
receiving, at a human editor layer, an adjustment to the second modified transcript from at least one human editor that is used to output a final transcript that wherein the final transcript is used to fine-tune the data-driven, statistical layer.

13. The method of claim 12 , wherein the human editor layer is associated with at least one of: (i) multiple-level human labeling, (ii) pairwise human labeling, and (iii) manual human transcription.

14. The method of claim 13 , wherein the adjustment is associated with at least one of: (i) syntactic correctness, (ii) semantic closeness, (iii) fluency, and (iv) style.

15. The method of claim 12 , wherein the final transcript is transmitted to a downstream task associated with at least one of: (i) language understanding, (ii) machine translation, (iii) text summarization, (iv) text classification, (v) information extraction, and (vi) question answering.

16. A non-transient, computer-readable medium storing instructions to be executed by a processor to perform a method for a multi-layer speech recognition transcript post processing system, the method comprising:
selecting, by a data-driven, statistical layer associated with a trained automatic speech recognition model, an initial transcript;
receiving, by a rule-based layer, the initial transcript and executing at least one pre-determined rule to generate a first modified transcript;
receiving, by a machine learning approach layer, the first modified transcript from the rule-based layer and performing a neural model inference to create a second modified transcript and
receiving, at a human editor layer, an adjustment to the second modified transcript from at least one human editor that is used to output a final transcript, wherein the final transcript is used to fine-tune the data-driven, statistical layer.

17. The medium of claim 16 , wherein the data-driven, statistical layer selects a best initial transcript from a set of N most probable speech recognition transcripts.

18. The medium of claim 16 , wherein the pre-determined rule is associated with at least one of: (i) a white list, (ii) a black list, and (iii) a rule approach.

19. The medium of claim 16 , wherein the machine learning approach layer is associated with at least one of: (i) online candidate generation, (ii) online neural model inference encoding and decoding, and (iii) online ranking.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Dimitriadis Dimitrios Basile
• Chen Xie
• Zeng Nanshan
• Shi Yu
• Lu Liyang

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G10L15/1815

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G10L2015/221

  Announcement of recognition results · CPC title