Speaker attributed transcript generation

Patent metadata

Field	Value
Publication number	US-11322148-B2
Application number	US-201916399166-A
Country	US
Kind code	B2
Filing date	Apr 30, 2019
Priority date	Apr 30, 2019
Publication date	May 3, 2022
Grant date	May 3, 2022

Abstract

A computer implemented method processes audio streams recorded during a meeting by a plurality of distributed devices. Operations include performing speech recognition on each audio stream by a corresponding speech recognition system to generate utterance-level posterior probabilities as hypotheses for each audio stream, aligning the hypotheses and formatting them as word confusion networks with associated word-level posteriors probabilities, performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses, formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed hypotheses for each audio stream as a speaker confusion network, aligning the word and speaker confusion networks from all audio streams to each other to merge the posterior probabilities and align word and speaker labels, and creating a best speaker-attributed word transcript by selecting the sequence of word and speaker labels with the highest posterior probabilities.

First claim

The invention claimed is:

1. A computer implemented method of processing audio streams captured during a meeting by a plurality of distributed devices includes operations performed by one or more processors, the operations comprising:
performing speech recognition on each audio stream by a corresponding speech recognition system to generate utterance-level posterior probabilities as hypotheses for each audio stream;
aligning the hypotheses and formatting them as word confusion networks with associated word-level posteriors probabilities;
performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses, wherein the speaker and/or word hypotheses streams originate from multiple partial combination of K out of N of the audio streams, where K<N;
formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed hypotheses for each audio stream as a speaker confusion network;
aligning the word and speaker confusion networks from all audio streams to each other to merge the posterior probabilities and align word and speaker labels; and
creating a best speaker-attributed word transcript by reading off the sequence of word and speaker labels with the highest posterior probabilities.

2. The method of claim 1 wherein the operations are performed on successive time windows applied to the audio streams such that the processing is performed incrementally to enable production of the speaker-attributed word recognition hypotheses in real-time.

3. The method of claim 2 , wherein the speaker and/or word hypotheses streams originate from multiple partial combination of input audio streams via fusion of the K out of N audio streams.

4. The method of claim 2 , wherein the speaker and/or word hypotheses streams originate from multiple partial combination of acoustic models applied to the K out of N audio streams, which in turn result from raw audio signals or fusion of the K out of N audio signals.

5. The method of claim 2 wherein the word and speaker hypotheses are truncated in time to a common time window applied to all audio streams based on the time marks associated with the word hypotheses generated for each audio stream.

6. The method of claim 5 , wherein the combination of K out of N raw audio streams is based on audio-quality criteria and/or based on the relative position of the speakers with respect to the distributed devices.

7. The method of claim 1 , wherein the speaker and/or word hypotheses streams originate from multiple partial combination of input audio streams via fusion of the K out of N audio streams.

8. The method of claim 4 , wherein the combination of K out of N acoustic model outputs is based on audio-quality criteria of the input signals and/or based on the relative position of the speakers with respect to the distributed devices.

9. The method of claim 1 , wherein the input speaker and/or word hypotheses streams originate from multiple partial combinations of acoustic models applied to the K out of N audio streams, which in turn result from raw audio streams or fusion of the K out of N audio streams.

10. A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method of processing audio streams captured during a meeting by a plurality of distributed devices, the operations comprising:
performing speech recognition on each audio stream by a corresponding speech recognition system to generate utterance-level posterior probabilities as hypotheses for each audio stream;
aligning the hypotheses and formatting them as word confusion networks with associated word-level posteriors probabilities;
performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses, wherein the speaker and/or word hypotheses streams originate from multiple partial combination of K out of N of the audio streams, where K<N;
formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed hypotheses for each audio stream as a speaker confusion network;
aligning the word and speaker confusion networks from all audio streams to each other to merge the posterior probabilities and align word and speaker labels; and
creating a best speaker-attributed word transcript by reading off the sequence of word and speaker labels with the highest posterior probabilities.

11. The method of claim 10 wherein the operations are performed on successive time windows applied to the audio streams such that the processing is performed incrementally to enable production of the speaker-attributed word recognition hypotheses in real-time.

12. The method of claim 10 , wherein the speaker and/or word hypotheses streams originate from multiple partial combination of input audio streams via fusion of the K out of N audio streams.

13. The method of claim 11 , wherein the speaker and/or word hypotheses streams originate from multiple partial combination of acoustic models applied to the K out of N audio streams, which in turn result from raw audio signals or fusion of the K out of N audio signals.

14. The method of claim 11 wherein the input hypotheses are truncated in time to a common time window applied to all audio streams based on the time marks associated with the word hypotheses generated for each audio stream.

15. The method of claim 14 , wherein the combination of K out of N raw audio streams is based on audio-quality criteria and/or based on the relative position of the speakers with respect to the distributed devices.

16. The method of claim 10 , wherein the input speaker and/or word hypotheses streams originate from multiple partial combination of input audio streams via fusion of the K out of N audio streams.

17. A device comprising:
a processor; and
a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations to process audio streams captured during a meeting by a plurality of distributed devices, the operations comprising:
performing speech recognition on each audio stream by a corresponding speech recognition system to generate utterance-level posterior probabilities as hypotheses for each audio stream;
aligning the hypotheses and formatting them as word confusion networks with associated word-level posteriors probabilities;
performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses, wherein the speaker and/or word hypotheses streams originate from multiple partial combination of K out of N of the audio streams, where K<N;
formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed hypotheses for each audio stream as a speaker confusion network;
aligning the word and speaker confusion networks from all audio streams to each other to merge the posterior probabilities and align word and speaker labels; and
creating a best speaker-attributed word transcript by reading off the sequence of word and speaker labels with the highest posterior probabilities.

18. The device of claim 17 wherein the operations are performed on successive time windows applied to the audio streams such that the processing is performed incrementally to enable production of the speaker-attributed word recognition hypotheses in real-time.

19. The d

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Yoshioka Takuya
• Stolcke Andreas
• Chen Zhuo
• Dimitriadis Dimitrios Basile
• Zeng Nanshan
• Qin Lijuan
• Hinthorn William Isaac
• Huang Xuedong

Classifications

• G10L15/08

  Speech classification or search · CPC title

• G10L15/26 Primary

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G10L19/018

  Audio watermarking, i.e. embedding inaudible data in the audio signal · CPC title

• H04L12/1831 Primary

  Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status · CPC title

• H04L12/1822

  Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission · CPC title