Neural machine translation systems with rare word processing

Patent metadata

Field	Value
Publication number	US-10133739-B2
Application number	US-201514921925-A
Country	US
Kind code	B2
Filing date	Oct 23, 2015
Priority date	Oct 24, 2014
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural translation systems with rare word processing. One of the methods is a method training a neural network translation system to track the source in source sentences of unknown words in target sentences, in a source language and a target language, respectively and includes deriving alignment data from a parallel corpus, the alignment data identifying, in each pair of source and target language sentences in the parallel corpus, aligned source and target words; annotating the sentences in the parallel corpus according to the alignment data and a rare word model to generate a training dataset of paired source and target language sentences; and training a neural network translation model on the training dataset.

First claim

What is claimed is:

1. A computer-implemented translation system for translating natural language text from a source sentence in a source language to a target sentence in a target language, the translation system comprising one or more computers and one or more storage devices storing translation instructions and translation data, wherein:
the translation data includes:
a word dictionary; and
a neural network translation model trained to process a sequence of inputs corresponding to words in the source sentence thereby generating a sequence of outputs corresponding to words in the target sentence, including emitting a respective unknown token in the sequence of outputs for each out-of-vocabulary (OOV) word that occurs in the target sentence, the model being operable to emit pointer tokens as a first type of unknown token and null tokens as a second type of unknown token, wherein pointer tokens are unknown tokens that identify a respective source word in the source sentence that corresponds to the unknown token, and null tokens are tokens that do not identify any source word in the source sentence that corresponds to the unknown token; and

the translation instructions are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
for every pointer token in the target sentence emitted by the neural network translation model from the source sentence, replacing the pointer token according to the corresponding source word in the source sentence.

2. The translation system of claim 1 , wherein replacing every pointer token comprises, for each pointer token:
using the word dictionary to perform a word translation from the respective source word in the source sentence that is identified by the pointer token and replacing the pointer token with the result of the word translation, or, if there is no such translation in the word dictionary for the respective source word in the source sentence that is identified by the pointer token, replacing the pointer token in the target sentence with the respective source word from the source sentence.

3. The translation system of claim 1 , wherein:
the neural network translation model contains a deep encoder Long Short-Term Memory model and a deep decoder Long Short-Term Memory model, wherein:
the encoder is trained to be operable to read the source sentence, one word at a time, to produce a large hidden state that summarizes the entire source sentence; and
the decoder is initialized from a final hidden state of the encoder and is trained to be operable to generate a target translation, one word at a time, until the decoder emits an end-of-sentence symbol.

4. The translation system of claim 1 , wherein:
the neural network translation model is a deep neural network.

5. The translation system of claim 1 , wherein:
the neural network translation model is a large deep Long Short-Term Memory model.

6. The translation system of claim 1 , wherein:
the neural network translation model is a six-layer deep Long Short-Term Memory model.

7. A method performed by one or more computers of a translation system, comprising:
providing, to a neural network translation model, a sequence of inputs corresponding to words of a source sentence in a source language;
obtaining, from the neural network translation model, as a result of the neural network translation model processing the sequence of inputs, a sequence of outputs corresponding to words of a target sentence in a target language,
wherein the sequence of outputs includes a pointer token that represents an out-of-vocabulary (OOV) word that occurs in the target sentence,
wherein the pointer token identifies a word from the source sentence that corresponds to the OOV word in the target sentence;

determining whether a translation in the target language is available in a word dictionary for the word from the source sentence that corresponds to the OOV word in the target sentence; and
in response to determining that a translation in the target language is available in the word dictionary for the word from the source sentence that corresponds to the OOV word in the target sentence, replacing the pointer token with the translation for the word from the source sentence that corresponds to the OOV word in the target sentence.

8. The method of claim 7 , wherein:
the sequence of outputs further includes a null token that represents a second OOV word that occurs in the target sentence, and the null token does not identify any word from the source sentence that corresponds to the second OOV word in the target sentence.

9. The method of claim 7 , wherein the sequence of outputs further includes a second pointer token that represents a second OOV word that occurs in the target sentence, and the second pointer token identifies a second word from the source sentence that corresponds to the second OOV word in the target sentence;
wherein the method further comprises:
determining whether a translation in the target language is available in the word dictionary for the second word from the source sentence that corresponds to the second OOV word in the target sentence; and
in response to determining that a translation in the target language is not available in the word dictionary for the second word from the source sentence that corresponds to the second OOV word in the target sentence, replacing the second pointer token in the target sentence with the second word from the source sentence in the source language.

10. The method of claim 7 , wherein:
the neural network translation model is a deep neural network.

11. The method of claim 7 , wherein:
the neural network translation model is a large deep Long Short-Term Memory model.

12. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
providing, to a neural network translation model, a sequence of inputs corresponding to words of a source sentence in a source language;
obtaining, from the neural network translation model, as a result of the neural network translation model processing the sequence of inputs, a sequence of outputs corresponding to words of a target sentence in a target language,
wherein the sequence of outputs includes a pointer token that represents an out-of-vocabulary (OOV) word that occurs in the target sentence,
wherein the pointer token identifies a word from the source sentence that corresponds to the OOV word in the target sentence;

determining whether a translation in the target language is available in a word dictionary for the word from the source sentence that corresponds to the OOV word in the target sentence; and
in response to determining that a translation in the target language is available in the word dictionary for the word from the source sentence that corresponds to the OOV word in the target sentence, replacing the pointer token with the translation for the word from the source sentence that corresponds to the OOV word in the target sentence.

13. The computer-readable medium of claim 12 , wherein:
the sequence of outputs further includes a null token that represents a second OOV word that occurs in the target sentence, and the null token does not identify any word from the source sentence that corresponds to the second OOV word in the target sentence.

14. The computer-readable medium of claim 12 , wherein the sequence of outputs further includes a second pointer token that represents a second OOV word that occurs in the target sentence, and the second pointer token ide

Assignees

• Google Llc

Inventors

• Le Quoc V
• Luong Minh-Thang
• Sutskever Ilya
• Vinyals Oriol
• Zaremba Wojciech

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F7/10

  Selecting, i.e. obtaining data of one kind from those record carriers which are identifiable by data of a second kind from a mass of ordered or randomly- distributed record carriers · CPC title

• G06F40/45

  Example-based machine translation; Alignment · CPC title

• G10L15/16

  using artificial neural networks · CPC title