Processing overlapping speech from distributed devices

Patent metadata

Field	Value
Publication number	US-11138980-B2
Application number	US-201916399175-A
Country	US
Kind code	B2
Filing date	Apr 30, 2019
Priority date	Apr 30, 2019
Publication date	Oct 5, 2021
Grant date	Oct 5, 2021

Abstract

A computer implemented method includes receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices, performing, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech, and providing the separated speech on a fixed number of separate output audio channels.

First claim

The invention claimed is:

1. A computer implemented method comprising:
receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices;
detecting overlapped speech during a first period of time during a meeting;
detecting no overlapped speech during a second period of time during the meeting;
performing for the first period of time, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech in response to detecting the overlapped speech, wherein the neural network model comprises a local observer comprising a set of stacked attention layers that map each audio signal into a representation;
providing the separated speech on a fixed number of separate output audio channels; and
providing the nonoverlapped speech for the second period of time on a further output audio channel without performing continuous speech separation.

2. The method of claim 1 wherein performing continuous speech separation is performed by the neural network model trained using permutation invariant training.

3. The method of claim 2 wherein the neural network model is configured to receive a varying number of inputs to support a dynamic change in a number of audio signals and locations of distributed devices during a meeting between multiple users.

4. The method of claim 1 wherein the multiple devices capture the audio signals during an ad-hoc meeting.

5. The method of claim 1 wherein the audio signals are received at a meeting server coupled to the distributed devices via a network.

6. The method of claim 1 and further comprising generating a transcript based on the separate audio channels.

7. The method of claim 6 and further comprising including speaker attribution in the generated transcript.

8. The method of claim 7 and further comprising sending the transcript to one or more of the distributed devices.

9. The method of claim 1 wherein at least two of the audio streams are provided by an ambient capture device having an array of microphones in fixed positions.

10. A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method, the operations comprising:
receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices;
detecting overlapped speech during a first period of time during a meeting;
detecting no overlapped speech during a second period of time during the meeting;
performing for the first period of time, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech in response to detecting the overlapped speech, wherein the neural network model comprises a local observer comprising a set of stacked attention layers that map each audio signal into a representation;
providing the separated speech on a fixed number of separate output audio channels; and
providing the nonoverlapped speech for the second period of time on a further output audio channel without performing continuous speech separation.

11. The device of claim 10 wherein performing continuous speech separation is performed by a neural network model trained using permutation invariant training.

12. The device of claim 11 wherein the neural network model is configured to receive a varying number of inputs to support a dynamic change in a number of audio signals and locations of distributed devices during a meeting between multiple users.

13. The device of claim 10 wherein the multiple distributed devices comprise wireless devices associated with speakers in a meeting.

14. The device of claim 10 wherein the audio signals are received at a meeting server coupled to the distributed devices via a network.

15. The device of claim 10 and further comprising generating a speaker attributed transcript based on the separate audio channels.

16. The device of claim 15 and further comprising sending the transcript to one or more of the distributed devices.

17. The device of claim 10 wherein at least two of the audio streams are provided by an ambient capture device having an array of microphones in fixed positions.

18. A device comprising:
a processor; and
a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations comprising:
receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices;
detecting overlapped speech during a first period of time during a meeting;
detecting no overlapped speech during a second period of time during the meeting;
performing for the first period of time, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech in response to detecting the overlapped speech, wherein the neural network model comprises a local observer comprising a set of stacked attention layers that map each audio signal into a representation;
providing the separated speech on a fixed number of separate output audio channels; and
providing the nonoverlapped speech for the second period of time on a further output audio channel without performing continuous speech separation.

19. The device of claim 18 wherein performing continuous speech separation is performed by a neural network model trained using permutation invariant training and wherein the neural network model is configured to receive a varying number of inputs to support a dynamic change in a number of audio signals and locations of distributed devices during a meeting between multiple users.

20. The device of claim 18 wherein the audio signals are received at a meeting server coupled to the distributed devices via a network, and wherein the meeting server performs addition operations comprising:
generating a speaker attributed transcript based on the separate audio channels; and
sending the transcript to one or more of the distributed devices.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Yoshioka Takuya
• Stolcke Andreas
• Chen Zhuo
• Dimitriadis Dimitrios Basile
• Zeng Nanshan
• Qin Lijuan
• Hinthorn William Isaac
• Huang Xuedong

Classifications

• G10L21/0208

  Noise filtering · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G10L15/30 Primary

  Distributed recognition, e.g. in client-server systems, for mobile phones or network applications · CPC title

• G10L21/0272 Primary

  Voice signal separating · CPC title

• G10L2021/02087

  the noise being separate speech, e.g. cocktail party · CPC title