Transcription knowledge graph

Patent metadata

Field	Value
Publication number	US-12431123-B2
Application number	US-202318332410-A
Country	US
Kind code	B2
Filing date	Jun 9, 2023
Priority date	Jun 9, 2023
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

Some embodiments include a transcription knowledge graph that can resolve automatic speech recognition (ASR) engine output errors. In some embodiments, a transcription knowledge graph can utilize data from past sessions of the ASR engine to form a voice graph that can be analyzed to determine a correlation between a mis-transcription (error text) and the correct transcription (correct text). Thus, ASR engine outputs, even if they include a mis-transcription, can be adjusted to the correct transcription. Further, the correct transcriptions and the voice graph can be used to train machine learning (ML) algorithms to generate numerical representations of an entity. The ML algorithms can be applied to a transcription to correctly identify a corresponding entity label, even if the transcription was not utilized in the voice graph to train the ML algorithm.

First claim

What is claimed is:

1. A computer-implemented method for correcting automatic speech recognition (ASR) engine output, comprising:
receiving, by at least one computer processor, a transcription comprising media content, wherein the transcription is generated via an ASR engine;
generating a voice graph based at least on previous ASR transcriptions of n-best outputs, where n is an integer,
wherein the voice graph comprises n nodes and at least (n−1) edges, wherein a first node of the n nodes corresponds to a top-1 transcript, and an n th node corresponds to a top-n transcript, where n>=2, and wherein an (n−1) edge of the at least (n−1) edges corresponds to the first node and the n th node;
selecting a candidate mined pair based at least on the voice graph, wherein the candidate mined pair comprises a mis-transcription and a correct transcription;
determining that the transcription corresponds to the mis-transcription; and
replacing the transcription with the correct transcription.

2. The computer-implemented method of claim 1 , wherein an attribute of the first node comprises: a frequency, a ranking distribution, or an associated entity.

3. The computer-implemented method of claim 1 , wherein an attribute of the (n−1) edge comprises: a co-occurrence frequency of the first node and the n th node, and a relatedness score.

4. The computer-implemented method of claim 3 , wherein the relatedness score comprises a pointwise mutual information (PMI) score.

5. The computer-implemented method of claim 1 , further comprising:
training a phoneme embedding generator with a plurality of candidate mined pairs including the candidate mined pair; and
generating a first vector representation of the media content using the phoneme embedding generator.

6. The computer-implemented method of claim 5 , further comprising:
generating a second vector representation of the transcription using the phoneme embedding generator;
determining that the first vector representation is more similar to the second vector representation than vector representations of other media content; and
selecting the media content, responsive to the determination of the first vector representation being more similar to the second vector representation.

7. The computer-implemented method of claim 1 , further comprising:
training a phoneme embedding generator with a plurality of candidate mined pairs excluding the candidate mined pair; and
generating a first vector representation of the media content using the phoneme embedding generator.

8. The computer-implemented method of claim 7 , further comprising:
generating a second vector representation of the transcription using the phoneme embedding generator;
determining that the first vector representation is more similar to the second vector representation than vector representations of other media content; and
selecting the media content, responsive to the determination of the first vector representation being more similar to the second vector representation.

9. A non-transitory computer-readable medium storing instructions that, when executed by a processor of a first electronic device, cause the first electronic device to perform operations, the operations comprising:
receiving a transcription comprising media content, wherein the transcription is generated via an automatic speech recognition (ASR) engine;
generating a voice graph based at least on previous ASR transcriptions of n-best outputs, where n is an integer,
wherein the voice graph comprises n nodes and at least (n−1) edges, wherein a first node of the n nodes corresponds to a top-1 transcript, and an n th node corresponds to a top-n transcript, where n>=2, and wherein an (n−1) edge of the at least (n−1) edges corresponds to the first node and the n th node;
selecting a candidate mined pair based at least on the voice graph, wherein the candidate mined pair comprises a mis-transcription and a correct transcription;
determining that the transcription corresponds to the mis-transcription; and
replacing the transcription with the correct transcription.

10. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise:
training a phoneme embedding generator with a plurality of candidate mined pairs including the candidate mined pair; and
generating a first vector representation of the media content using the phoneme embedding generator.

11. The non-transitory computer-readable medium of claim 10 , wherein the operations further comprise:
generating a second vector representation of the transcription using the phoneme embedding generator;
determining that the first vector representation is more similar to the second vector representation than vector representations of other media content; and
selecting the media content, responsive to the determination of the first vector representation being more similar to the second vector representation.

12. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise:
training a phoneme embedding generator with a plurality of candidate mined pairs excluding the candidate mined pair; and
generating a first vector representation of the media content using the phoneme embedding generator.

13. The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:
generating a second vector representation of the transcription using the phoneme embedding generator;
determining that the first vector representation is more similar to the second vector representation than vector representations of other media content; and
selecting the media content, responsive to the determination of the first vector representation being more similar to the second vector representation.

14. A system, comprising:
one or more memories; and
at least one processor each coupled to at least one of the memories and configured to perform operations comprising:
receiving a transcription comprising media content, wherein the transcription is generated via an automatic speech recognition (ASR) engine;
generating a voice graph based at least on previous ASR transcriptions of n-best outputs, where n is an integer,
wherein the voice graph comprises n nodes and at least (n−1) edges, wherein a first node of the n nodes corresponds to a top-1 transcript, and an n th node corresponds to a top-n transcript, where n>=2, and wherein an (n−1) edge of the at least (n−1) edges corresponds to the first node and the n th node;
selecting a candidate mined pair based at least on the voice graph, wherein the candidate mined pair comprises a mis-transcription and a correct transcription;
determining that the transcription corresponds to the mis-transcription; and
replacing the transcription with the correct transcription.

15. The system of claim 14 , wherein the operations further comprise:
training a phoneme embedding generator with a plurality of candidate mined pairs including the candidate mined pair; and
generating a first vector representation of the media content using the phoneme embedding generator.

16. The system of claim 15 , wherein the operations further comprise:
generating a second vector representation of the transcription using the phoneme embedding generator;
determining that the first vector representation is more similar to the second vector representation than vector representations of other media content; and
selecting the media content, responsive to the determination of the first vector representation being more similar to the second vector representation.

Assignees

• Roku Inc

Inventors

• Kim Doo Soon
• Heo Minsuk
• Yeh Zei-Chan
• Asefisaray Behnam
• Mangalath Praful Chandra

Classifications

• G10L15/26 Primary

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G10L2015/025

  Phonemes, fenemes or fenones being the recognition units · CPC title

• G10L2015/0633

  using lexical or orthographic knowledge sources · CPC title

• G10L15/02

  Feature extraction for speech recognition; Selection of recognition unit · CPC title

• G10L15/063 Primary

  Training · CPC title