Multi-person speech separation method and apparatus using a generative adversarial network model

Patent metadata

Field	Value
Publication number	US-11450337-B2
Application number	US-202017023829-A
Country	US
Kind code	B2
Filing date	Sep 17, 2020
Priority date	Aug 9, 2018
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

A multi-person speech separation method is provided for a terminal. The method includes extracting a hybrid speech feature from a hybrid speech signal requiring separation, N human voices being mixed in the hybrid speech signal, N being a positive integer greater than or equal to 2; extracting a masking coefficient of the hybrid speech feature by using a generative adversarial network (GAN) model, to obtain a masking matrix corresponding to the N human voices, wherein the GAN model comprises a generative network model and an adversarial network model; and performing a speech separation on the masking matrix corresponding to the N human voices and the hybrid speech signal by using the GAN model, and outputting N separated speech signals corresponding to the N human voices.

First claim

What is claimed is:

1. A multi-person speech separation method for a terminal by using a generative adversarial network (GAN) model, the GAN model including a generative network model and a discriminative network model, the method comprising:
obtaining a hybrid speech sample and a clean speech sample from a sample database;
extracting a hybrid speech sample feature from the hybrid speech sample;
extracting a masking coefficient of the hybrid speech sample feature by using the generative network model, to obtain a sample masking matrix;
performing a speech separation on the sample masking matrix and the hybrid speech sample by using the generative network model, and outputting a separated speech sample;
performing alternate training on the generative network model and the discriminative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample, wherein the alternate training is performed by:
fixing the generative network model during a current time of training of the discriminative network model;
obtaining a loss function of the discriminative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample;
optimizing the discriminative network model by minimizing the loss function of the discriminative network model;
fixing the discriminative network model during a next time of training of the generative network model;
obtaining a loss function of the generative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample; and
optimizing the generative network model by minimizing the loss function of the generative network model.

2. The method according to claim 1 , wherein the obtaining a loss function of the discriminative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample comprises:
determining a first signal sample combination according to the separated speech sample and the hybrid speech sample, and determining a second signal sample combination according to the clean speech sample and the hybrid speech sample;
performing discriminative output on the first signal sample combination by using the discriminative network model to obtain a first discriminative output result, and obtaining a first distortion metric between the first discriminative output result and a first target output of the discriminative network model;
performing discriminative output on the second signal sample combination by using the discriminative network model to obtain a second discriminative output result, and obtaining a second distortion metric between the second discriminative output result and a second target output of the discriminative network model; and
obtaining the loss function of the discriminative network model according to the first distortion metric and the second distortion metric.

3. The method according to claim 1 , wherein the extracting a hybrid speech feature from a hybrid speech signal requiring separation comprises:
extracting a time domain feature or a frequency domain feature of a single-channel speech signal from the hybrid speech signal;
extracting a time domain feature or a frequency domain feature of a multi-channel speech signal from the hybrid speech signal;
extracting a single-channel speech feature from the hybrid speech signal; or
extracting a correlated feature among a plurality of channels from the hybrid speech signal.

4. The method according to claim 1 , wherein the obtaining a loss function of the generative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample comprises:
determining a first signal sample combination according to the separated speech sample and the hybrid speech sample;
performing discriminative output on the first signal sample combination by using the discriminative network model to obtain a first discriminative output result, and obtaining a third distortion metric between the first discriminative output result and a second target output of the discriminative network model;
obtaining a fourth distortion metric between the separated speech sample and clean speech; and
obtaining the loss function of the generative network model according to the third distortion metric and the fourth distortion metric.

5. The method according to claim 4 , wherein the obtaining a fourth distortion metric between the separated speech sample and clean speech comprises:
performing a permutation invariant calculation on the separated speech sample and the clean speech sample to obtain a correspondence result between the separated speech sample and the clean speech sample; and
obtaining the fourth distortion metric according to the correspondence result between the separated speech sample and the clean speech sample.

6. The multi-person speech separation method according to claim 1 , further comprising:
performing a speech separation on a hybrid speech signal by using the GAN model, the hybrid speed signal including N human voices, N being a positive integer greater than or equal to claim 2 .

7. The multi-person speech separation method according to claim 6 , wherein performing the speech separation on the hybrid speech signal by using the GAN model comprises:
extracting a hybrid speech feature from the hybrid speech signal;
extracting a masking coefficient of the hybrid speech feature to obtain a masking matrix corresponding to the N human voices; and
performing the speech separation on the masking matrix and outputting N separated speech signals corresponding to the N human voices.

8. A multi-person speech separation apparatus, comprising: a memory storing computer program instructions; and a processor coupled to the memory and, when executing the computer program instructions, configured to perform a multi-person speech separation method by using a generative adversarial network (GAN) model, the GAN model including a generative network model and a discriminative network model, the method comprising:
obtaining a hybrid speech sample and a clean speech sample from a sample database;
extracting a hybrid speech sample feature from the hybrid speech sample;
extracting a masking coefficient of the hybrid speech sample feature by using the generative network model, to obtain a sample masking matrix;
performing a speech separation on the sample masking matrix and the hybrid speech sample by using the generative network model, and outputting a separated speech sample;
performing alternate training on the generative network model and the discriminative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample, wherein the alternate training is performed by:
fixing the generative network model during a current time of training of the discriminative network model;
obtaining a loss function of the discriminative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample;
optimizing the discriminative network model by minimizing the loss function of the discriminative network model;
fixing the discriminative network model during a next time of training of the generative network model;
obtaining a loss function of the generative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample; and
optimizing the generative network model by minimizing the loss function of the generative network model.

9. The apparatus according to claim 8 , wherein the obtaining a loss function of the discriminative network model by using the separated speech sample, the hybrid speech sample, and the clean speech sample comprises:

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Chen Lianwu
• Yu Meng
• Qian Yanmin
• Su Dan
• Yu Dong

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title