Multi-microphone speech separation

Patent metadata

Field	Value
Publication number	US-10957337-B2
Application number	US-201815991988-A
Country	US
Kind code	B2
Filing date	May 29, 2018
Priority date	Apr 11, 2018
Publication date	Mar 23, 2021
Grant date	Mar 23, 2021

Abstract

This document relates to separation of audio signals into speaker-specific signals. One example obtains features reflecting mixed speech signals captured by multiple microphones. The features can be input a neural network and masks can be obtained from the neural network. The masks can be applied one or more of the mixed speech signals captured by one or more of the microphones to obtain two or more separate speaker-specific speech signals, which can then be output.

First claim

The invention claimed is:

1. A method performed on a computing device, the method comprising:
obtaining features reflecting mixed speech signals captured by multiple microphones;
inputting the features to a neural network;
obtaining masks output by the neural network;
applying the masks to at least one of the mixed speech signals captured by at least one of the microphones to obtain two or more separate speaker-specific speech signals;
gain adjusting the two or more separate speaker-specific speech signals; and
outputting the two or more separate speaker-specific speech signals after the gain adjusting.

2. The method of claim 1 , further comprising:
applying the masks by deriving beamformers from the masks and using the beamformers to obtain the two or more separate speaker-specific speech signals.

3. The method of claim 1 , further comprising:
training the neural network using permutation invariant training.

4. The method of claim 1 , the features comprising power spectra and phase features.

5. The method of claim 1 , further comprising:
normalizing the features before inputting the features to the neural network.

6. The method of claim 1 , further comprising:
inputting the features to the neural network as sliding windows, individual sliding windows comprising multiple audio segments;
obtaining the masks from the neural network, the masks being output for respective sliding windows; and
stitching masks for at least two adjacent sliding windows together.

7. The method of claim 6 , further comprising:
processing the respective mixed speech signals to obtain the features, the features comprising plurality of frequency bins for individual segments of audio from each microphone, each frequency bin comprising magnitude and phase values of a Fourier transform.

8. The method of claim 7 , further comprising:
combining the individual segments into the sliding windows, the sliding windows having respective overlapping portions.

9. A system comprising:
a hardware processing unit; and
a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to:
obtain features reflecting multiple mixed speech signals captured by multiple microphones;
normalize the features to obtain normalized features;
input the normalized features to a speech separation model;
obtain, from the speech separation model, respective masks for individual speakers; and
apply the respective masks to at least one of the mixed speech signals to obtain at least two separate speaker-specific speech signals for different speakers.

10. The system of claim 9 , further comprising the multiple microphones.

11. The system of claim 10 , the multiple microphones being synchronized via a shared clock, wherein the shared clock is a local hardware clock or a logical clock synchronized over a network.

12. The system of claim 9 , wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
perform automated speech recognition on the at least two separate speaker-specific speech signals to identify words spoken by the different speakers.

13. The system of claim 12 , wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
produce a transcript identifying first words spoken by a first speaker and second words spoken by a second speaker.

14. The system of claim 12 , wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
perform a first action in response to a first word spoken by a first speaker; and
perform a second action in response to a second word spoken by a second speaker.

15. The system of claim 9 , the features comprising power spectra and inter-microphone phase differences.

16. The system of claim 15 , wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
perform mean and variance normalization on the power spectra; and
perform mean normalization without variance normalization on the inter-microphone phase differences.

17. A method comprising:
obtaining features reflecting multiple mixed speech signals captured by multiple microphones;
normalizing the features to obtain normalized features;
inputting the normalized features to a speech separation model;
obtaining, from the speech separation model, respective masks for individual speakers; and
applying the respective masks to at least one of the mixed speech signals to obtain at least two separate speaker-specific speech signals for different speakers.

18. The method of claim 17 , performed in the absence of data indicating a number of speakers present in the multiple mixed speech signals.

19. The method of claim 18 , performed multiple times as the number of speakers present in the multiple mixed speech signals changes over time.

20. The method of claim 17 , further comprising:
performing a dereverberation process on the multiple mixed speech signals prior to inputting the normalized features to the speech separation model.

21. The method of claim 20 , wherein the dereverberation process comprises applying a dereverberation filter to the multiple mixed speech signals prior to obtaining the features.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chen Zhuo
• Erdogan Hakan
• Yoshioka Takuya
• Alleva Fileno A
• Xiao Xiong

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G10L15/16

  using artificial neural networks · CPC title