N-best softmax smoothing for minimum bayes risk training of attention based sequence-to-sequence models

Patent metadata

Field	Value
Publication number	US-11803618-B2
Application number	US-202217989536-A
Country	US
Kind code	B2
Filing date	Nov 17, 2022
Priority date	Nov 14, 2018
Publication date	Oct 31, 2023
Grant date	Oct 31, 2023

Abstract

A method and apparatus are provided that analyzing sequence-to-sequence data, such as sequence-to-sequence speech data or sequence-to-sequence machine translation data for example, by minimum Bayes risk (MBR) training a sequence-to-sequence model and within introduction of applications of softmax smoothing to an N-best generation of the MBR training of the sequence-to-sequence model.

First claim

What is claimed is:

1. An apparatus comprising:
at least one memory configured to store computer program code;
at least one hardware processor configured to access said computer program code and operate as instructed by said computer program code, said computer program code including:
training code configured to cause said at least one hardware processor to train a sequence-to-sequence model; and
smoothing code configured to cause said at least one hardware processor to, during an N-best generation of the training, apply softmax smoothing to a label prediction distribution and generate scores at steps of a beam-search of the N-best generation of the training.

2. The apparatus according to claim 1 , wherein the computer program code further includes beam search code configured to cause said at least one hardware processor to perform the beam search during the training.

3. The apparatus according to claim 2 , wherein the beam search code is further configured to apply the softmax smoothing to the label prediction distribution at each iteration of the beam search.

4. The apparatus according to claim 3 , wherein the computer program code further comprises obtaining code configured to cause said at least one processor to obtain hypothesized outputs applied to a hypothesis space for the training.

5. The apparatus according to claim 1 , wherein the training code is configured to cause said at least one processor to apply a loss operation to the training data and corresponding reference label sequences.

6. The apparatus according to claim 5 , wherein the training data comprises speech data.

7. The apparatus according to claim 5 , wherein the training data comprises machine translation data.

8. The apparatus according to claim 5 , wherein the loss operation comprises a risk operation between a hypothesized label sequence and the reference label sequences.

9. The apparatus according to claim 5 , wherein the loss operation comprises a sequence probability given the training data.

10. The apparatus according to claim 8 , wherein the training further comprises deriving a gradient of the loss operation with respect to a probability, regarding a particular label of the label prediction distribution, and the risk operation.

11. A method performed by at least one computer processor comprising:
training a sequence-to-sequence model; and
during an N-best generation of the training, applying softmax smoothing to a label prediction distribution and generating scores at steps of a beam search of the N-best generation of the training.

12. The method according to claim 11 , further comprising:
performing the beam search during the training.

13. The method according to claim 12 , further comprising:
during each step of the beam search, applying the softmax smoothing to the label prediction distribution.

14. The method according to claim 13 , further comprising:
obtaining hypothesized outputs applied to a hypothesis space for the training.

15. The method according to claim 11 , wherein the training comprises applying a loss operation to the training data and corresponding reference label sequences.

16. The method according to claim 15 , wherein the training data comprises speech data.

17. The method according to claim 15 , wherein the training data comprises machine translation data.

18. The method according to claim 15 , wherein the loss operation comprises a risk operation between a hypothesized label sequence and the reference label sequences.

19. The method according to claim 15 , wherein the loss operation comprises a sequence probability given the training data.

20. A non-transitory computer readable medium storing a program causing a computer to execute a process, the process comprising:
training a sequence-to-sequence model; and
during an N-best generation of the training, applying softmax smoothing to label prediction distribution and generating scores at steps of a beam-search of the N-best generation of the training.

Assignees

• Tencent America LLC

Inventors

• Weng Chao
• Cui Jia
• Wang Guangsen
• Wang Jun
• Yu Chengzhu
• Su Dan
• Yu Dong

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F18/24155 Primary

  Bayesian classification · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title