Speaker attributed transcript generation

Patent metadata

Field	Value
Publication number	US-12243534-B2
Application number	US-202217712802-A
Country	US
Kind code	B2
Filing date	Apr 4, 2022
Priority date	Apr 30, 2019
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

A computer implemented method processes audio streams recorded during a meeting by a plurality of distributed devices. Operations include performing speech recognition on each audio stream by a corresponding speech recognition system to generate utterance-level posterior probabilities as hypotheses for each audio stream, aligning the hypotheses and formatting them as word confusion networks with associated word-level posteriors probabilities, performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses, formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed hypotheses for each audio stream as a speaker confusion network, aligning the word and speaker confusion networks from all audio streams to each other to merge the posterior probabilities and align word and speaker labels, and creating a best speaker-attributed word transcript by selecting the sequence of word and speaker labels with the highest posterior probabilities.

First claim

The invention claimed is:

1. A computer implemented method comprising:
receiving audio streams, each audio stream captured during a meeting by a different one of a plurality of distributed devices;
performing speech recognition on each audio stream by a corresponding speech recognition system;
performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses;
formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed word hypotheses for each audio stream;
aligning the word and speaker labels; and
creating a best speaker-attributed word transcript by identifying a sequence of word and speaker labels with the highest posterior probabilities wherein the speaker-attributed word hypotheses and speaker hypotheses are truncated in time to a common time window applied to all audio streams based on time marks associated with the word hypotheses generated for each audio stream.

2. The method of claim 1 wherein the operations are performed on successive time windows applied to the audio streams such that the processing is performed incrementally to enable production of the speaker-attributed word recognition hypotheses in real-time.

3. The method of claim 2 , wherein the combination of K out of N raw audio signals or fusion of the audio signals is based on audio-quality criteria and/or based on the relative position of the speakers with respect to the distributed devices.

4. The method of claim 3 , wherein the K out of N raw audio signals result from raw audio signals or a fusion of audio signals.

5. The method of claim 1 , wherein the speaker and/or word hypotheses streams originate from multiple partial combination of K out of N audio streams, where K<N.

6. The method of claim 5 , wherein the combination of K out of N acoustic model outputs is based on audio-quality criteria of the input signals and/or based on the relative position of the speakers with respect to the distributed devices.

7. The method of claim 1 , wherein the input speaker and/or word hypotheses streams originate from multiple partial combinations of acoustic models applied to K out of N audio streams where K<N, which in turn result from raw audio streams or fusion of audio streams.

8. The method of claim 1 , wherein the output of multiple acoustic models is applied to K out of N audio streams, where K<N, which in turn result from raw audio streams or fusion of audio streams that are combined as input to M speech recognition decoders.

9. A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method comprising:
receiving audio streams, each audio stream captured during a meeting by a different one of a plurality of distributed devices;
performing speech recognition on each audio stream by a corresponding speech recognition system;
performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses;
formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed word hypotheses for each audio stream;
aligning the word and speaker labels; and
creating a best speaker-attributed word transcript by identifying a sequence of word and speaker labels with the highest posterior probabilities wherein the speaker-attributed word hypotheses and speaker hypotheses are truncated in time to a common time window applied to all audio streams based on time marks associated with the word hypotheses generated for each audio stream.

10. The device of claim 9 wherein the operations are performed on successive time windows applied to the audio streams such that the processing is performed incrementally to enable production of the speaker-attributed word recognition hypotheses in real-time.

11. The device of claim 10 , wherein the combination of K out of N raw audio signals or fusion of the audio signals is based on audio-quality criteria and/or based on the relative position of the speakers with respect to the distributed devices.

12. The device of claim 11 , wherein the K out of N raw audio signals result from raw audio signals or a fusion of audio signals.

13. The device of claim 9 , wherein the speaker and/or word hypotheses streams originate from multiple partial combination of K out of N audio streams, where K<N.

14. The device of claim 13 , wherein the combination of K out of N acoustic model outputs is based on audio-quality criteria of the input signals and/or based on the relative position of the speakers with respect to the distributed devices.

15. The device of claim 9 , wherein the input speaker and/or word hypotheses streams originate from multiple partial combinations of acoustic models applied to K out of N audio streams where K<N, which in turn result from raw audio streams or fusion of audio streams.

16. A device comprising:
a processor; and
a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations comprising:
receiving audio streams, each audio stream captured during a meeting by a different one of a plurality of distributed devices;
performing speech recognition on each audio stream by a corresponding speech recognition system;
performing speaker recognition on each audio stream by a speaker identification algorithm that generates a stream of speaker-attributed word hypotheses;
formatting speaker hypotheses with associated speaker label posterior probabilities and speaker-attributed word hypotheses for each audio stream;
aligning the word and speaker labels; and
creating a best speaker-attributed word transcript by identifying a sequence of word and speaker labels with the highest posterior probabilities wherein the speaker-attributed word hypotheses and speaker hypotheses are truncated in time to a common time window applied to all audio streams based on time marks associated with the word hypotheses generated for each audio stream.

17. The device of claim 16 wherein the operations are performed on successive time windows applied to the audio streams such that the processing is performed incrementally to enable production of the speaker-attributed word recognition hypotheses in real-time.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Yoshioka Takuya
• Stolcke Andreas
• Chen Zhuo
• Dimitriadis Dimitrios Basile
• Zeng Nanshan
• Qin Lijuan
• Hinthorn William Isaac
• Huang Xuedong

Classifications

• G10L19/018

  Audio watermarking, i.e. embedding inaudible data in the audio signal · CPC title

• G10L15/08

  Speech classification or search · CPC title

• H04M3/42221

  Conversation recording systems (at the subscriber's set H04M1/656) · CPC title

• H04M2201/40

  using speech recognition · CPC title

• H04M2201/41

  using speaker recognition · CPC title