Adversarial teacher-student learning for unsupervised domain adaptation

Patent metadata

Field	Value
Publication number	US-10643602-B2
Application number	US-201815923795-A
Country	US
Kind code	B2
Filing date	Mar 16, 2018
Priority date	Mar 16, 2018
Publication date	May 5, 2020
Grant date	May 5, 2020

Abstract

Methods, systems, and computer programs are presented for training, with adversarial constraints, a student model for speech recognition based on a teacher model. One method includes operations for training a teacher model based on teacher speech data, initializing a student model with parameters obtained from the teacher model, and training the student model with adversarial teacher-student learning based on the teacher speech data and student speech data. Training the student model with adversarial teacher-student learning further includes minimizing a teacher-student loss that measures a divergence of outputs between the teacher model and the student model; minimizing a classifier condition loss with respect to parameters of a condition classifier; and maximizing the classifier condition loss with respect to parameters of a feature extractor. The classifier condition loss measures errors caused by acoustic condition classification. Further, speech is recognized with the trained student model.

First claim

What is claimed is:

1. A method comprising:
training, by one or more processors, a teacher model based on teacher speech data;
initializing, by the one or more processors, a student model with parameters obtained from the trained teacher model;
training, by the one or more processors, the student model with adversarial teacher-student learning based on the teacher speech data and student speech data, training the student model with adversarial teacher-student learning further comprising:
minimizing a teacher-student loss that measures a divergence of outputs between the teacher model and the student model;
minimizing a classifier condition loss with respect to parameters of a condition classifier, the classifier condition loss measuring errors caused by acoustic condition classification; and
maximizing the classifier condition loss with respect to parameters of a feature extractor; and

recognizing speech with the trained student model.

2. The method as recited in claim 1 , wherein the condition classifier is a neural network for mapping each deep feature to an acoustic condition.

3. The method as recited in claim 1 , wherein the divergence is a Kullback-Leibler divergence that measures how an output distribution of the teacher model diverges from an output distribution of the student model.

4. The method as recited in claim 1 , wherein the student model further comprises:
a classifier to classify units of speech, the units of speech being one of a senone, a phoneme, a tri-phone, a syllable, a character, a part of a word, or a word; and
a feature extractor for extracting deep features from the student speech data.

5. The method as recited in claim 1 , wherein the teacher speech data comprises a plurality of utterances in a teacher domain, wherein the student speech data comprises the plurality of utterances in a student domain, wherein training the student model further comprises:
providing the plurality of utterances from the teacher speech data in parallel to the plurality of utterances in the student speech data.

6. The method as recited in claim 1 , wherein the teacher-student loss is calculated by:
calculating a teacher senone posterior;
calculating a student senone posterior for a deep feature; and
calculating the teacher-student loss as a difference between the teacher senone posterior and the student senone posterior.

7. The method as recited in claim 1 , wherein a condition defines characteristics of a speaker and an environment where speech is captured.

8. The method as recited in claim 1 , wherein training the student model with adversarial teacher-student learning causes the student model to recognize senones similarly to how the teacher model recognizes senones in a condition-robust fashion.

9. The method as recited in claim 1 , wherein training the student model with adversarial teacher-student learning causes the student model to lack differentiation among different conditions.

10. The method as recited in claim 1 , wherein training the student model is performed iteratively by analyzing the teacher speech data and the student speech data.

11. A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
training a teacher model based on teacher speech data;
initializing a student model with parameters obtained from the trained teacher model;
training the student model with adversarial teacher-student learning based on the teacher speech data and student speech data, training the student model with adversarial teacher-student learning further comprising:
minimizing a teacher-student loss that measures a divergence of outputs between the teacher model and the student model;
minimizing a classifier condition loss with respect to parameters of a condition classifier, the classifier condition loss measuring errors caused by acoustic condition classification; and
maximizing the classifier condition loss with respect to parameters of a feature extractor; and

recognizing speech with the trained student model.

12. The system as recited in claim 11 , wherein the condition classifier is a neural network for mapping each deep feature to an acoustic condition.

13. The system as recited in claim 11 , wherein the student model further comprises:
a classifier to classify units of speech, the units of speech being one of a senone, a phoneme, a tri-phone, a syllable, a character, a part of a word, or a word; and
a feature extractor for extracting deep features from the student speech data.

14. The system as recited in claim 11 , wherein the teacher speech data comprises a plurality of utterances in a teacher domain, wherein the student speech data comprises the plurality of utterances in a student domain, wherein training the student model further comprises:
providing the plurality of utterances from the teacher speech data in parallel to the plurality of utterances in the student speech data.

15. The system as recited in claim 11 , wherein training the student model with adversarial teacher-student learning that causes the student model to recognize senones similarly to how the teacher model recognizes senones in a condition-robust fashion, wherein training the student model with adversarial teacher-student learning that cause the student model to lack differentiation among different conditions.

16. A machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
training a teacher model based on teacher speech data:
initializing a student model with parameters obtained from the trained teacher model;
training the student model with adversarial teacher-student learning based on the teacher speech data and student speech data, training the student model with adversarial teacher-student learning further comprising:
minimizing a teacher-student loss that measures a divergence of outputs between the teacher model and the student model;
minimizing a classifier condition loss with respect to parameters of a condition classifier, the classifier condition loss measuring errors caused by acoustic condition classification; and
maximizing the classifier condition loss with respect to parameters of a feature extractor; and

recognizing speech with the trained student model.

17. The machine-readable storage medium as recited in claim 16 , wherein the condition classifier is a neural network mapping each deep feature to an acoustic condition.

18. The machine-readable storage medium as recited in claim 16 , wherein the student model further comprises:
a classifier to classify units of speech, the units of speech being one of a senone, a phoneme, a tri-phone, a syllable, a character, a part of a word, or a word; and
a feature extractor for extracting deep features from the student speech data.

19. The machine-readable storage medium as recited in claim 16 , wherein the teacher speech data comprises a plurality of utterances in a teacher domain, wherein the student speech data comprises the plurality of utterances in a student domain, wherein training the student model further comprises:
providing the plurality of utterances from the teacher speech data in parallel to the plurality of utterances in the student speech data.

20. The machine-readable storage medium as recited in claim 16 , wherein training the student model wi

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Li Jinyu
• Meng Zhong
• Gong Yifan

Classifications

• G10L15/16

  using artificial neural networks · CPC title

• G06N20/00

  Machine learning · CPC title

• G10L15/063 Primary

  Training · CPC title

• G10L15/02

  Feature extraction for speech recognition; Selection of recognition unit · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title