Training sequence generation neural networks using quality scores

Patent metadata

Field	Value
Publication number	US-10540585-B2
Application number	US-201916421406-A
Country	US
Kind code	B2
Filing date	May 23, 2019
Priority date	May 23, 2018
Publication date	Jan 21, 2020
Grant date	Jan 21, 2020

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a sequence generation neural network. One of the methods includes obtaining a batch of training examples; for each of the training examples: processing the training network input in the training example using the neural network to generate an output sequence; for each particular output position in the output sequence: identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence, for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output, and determining an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score.

First claim

What is claimed is:

1. A method of training a neural network having a plurality of network parameters and being configured to map a system input to an output sequence comprising a plurality of system outputs, wherein the method comprises:
obtaining a batch of training examples, each training example comprising a training network input and, for each training network input, a ground truth output sequence;
for each of the training examples:
processing the training network input in the training example using the neural network and in accordance with current values of the network parameters to map the training network input to an output sequence comprising a respective system output from a vocabulary of possible system outputs at each of a plurality of output positions;
for each particular output position of the plurality of output positions in the output sequence:
identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence,
for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs, wherein the quality score measures a quality of a candidate output sequence relative to the ground truth output sequence, and
determining, using the highest quality scores for the possible system outputs, an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score; and

determining updated values of the network parameters from the updates for the particular output positions in the output sequences generated by the neural network for the batch of training examples.

2. The method of claim 1 , further comprising:
outputting the trained neural network for use in mapping new network inputs to new output sequences.

3. The method of claim 1 , wherein the neural network is configured to, for each of the output positions, generate a likelihood distribution over the possible system outputs in the vocabulary and select the system output at the output position using the likelihood distribution.

4. The method of claim 3 , wherein determining, using the highest quality scores for the possible system outputs, an update to the current values of the network parameters that increases a likelihood that the neural network generates a system output at the position that has a high quality score comprises:
generating a target likelihood distribution for the output position from the highest quality sores for the possible system outputs,
determining a gradient with respect to the network parameters of an objective function that depends on a divergence between the target likelihood distribution for the output position and the likelihood distribution generated by the neural network for the output position, and
determining the update to the current values using the gradient.

5. The method of claim 4 , wherein generating the target likelihood distribution comprises applying a softmax to the highest quality scores for the possible system outputs to generate a respective likelihood for each of the possible system outputs.

6. The method of claim 5 , wherein the softmax is applied with reduced temperature.

7. The method of claim 1 , wherein the possible system outputs in the vocabulary comprise tokens in a natural language.

8. The method of claim 1 , wherein the quality score is based on the edit distance between the candidate output sequence and the ground truth output sequence, and wherein the highest quality score that can be assigned is the quality score for the candidate output sequence that has a smallest edit distance to the ground truth output sequence.

9. The method of claim 1 , wherein determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs comprises:
determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by any ground truth suffix that is part of the ground truth output sequence;
identifying one or more ground truth prefixes of the ground truth output sequence relative to which the prefix has a highest quality score,
identifying, for each of the identified ground truth prefixes, a respective ground truth suffix that follows the identified ground truth prefix in the ground truth sequence, and
when the possible system output is a first system output in any of the identified ground truth suffixes, assigning as the highest quality score for the possible system output the highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by any ground truth suffix that is part of the ground truth output sequence.

10. The method of claim 9 , wherein determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs comprises:
determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by a possible system output that is not a first system output in any of the identified suffixes and followed by any ground truth suffix that is part of the ground truth output sequence;
when the possible system output is not the first system output in any of the identified ground truth suffixes, assigning as the highest quality score for the possible system output the highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by a possible system output that is not a first system output in any of the identified ground truth suffixes and followed by any ground truth suffix that is part of the ground truth output sequence.

11. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a neural network having a plurality of network parameters and being configured to map a system input to an output sequence comprising a plurality of system outputs, the operations comprising:
obtaining a batch of training examples, each training example comprising a training network input and, for each training network input, a ground truth output sequence;
for each of the training examples:
processing the training network input in the training example using the neural network and in accordance with current values of the network parameters to map the training network input to an output sequence comprising a respective system output from a vocabulary of possible system outputs at each of a plurality of output positions;
for each particular output position of the plurality of output positions in the output sequence:
identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence,
for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs, wherein the quality score measures a quality of a candidate output sequence relative to the ground truth output sequence, and
determining, using the highest quality scores for the possible system outputs, an update to the curren

Assignees

• Google Llc

Inventors

• Norouzi Mohammad
• Chan William
• Sabour Rouh Aghdam Sara

Classifications

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/048

  Activation functions · CPC title

• G10L25/30

  using neural networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title