Generating parse trees of text segments using neural networks

Patent metadata

Field	Value
Publication number	US-10409908-B2
Application number	US-201514976121-A
Country	US
Kind code	B2
Filing date	Dec 21, 2015
Priority date	Dec 19, 2014
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating parse trees for input text segments. One of the methods includes obtaining an input text segment, processing the input text segment using a first long short term memory (LSTM) neural network to convert the input text segment into an alternative representation for the input text segment, and processing the alternative representation for the input text segment using a second LSTM neural network to generate a linearized representation of a parse tree for the input text segment.

First claim

What is claimed is:

1. A method performed by one or more computers, the method comprising: obtaining an input text segment; processing the input text segment using a first long short term memory (LSTM) neural network to convert the input text segment into an alternative representation for the input text segment; and processing the alternative representation for the input text segment using a second LSTM neural network to generate a linearized representation of a hierarchical parse tree for the input text segment, including using the second LSTM neural network to sequentially select symbols for the linearized representation, the selected symbols including (i) first symbols that represent syntactic elements of the input text segment and (ii) second symbols that represent hierarchical relationships between particular first symbols in the hierarchical parse tree.

2. The method of claim 1 , further comprising: generating the hierarchical parse tree for the input text segment from the linearized representation of the hierarchical parse tree.

3. The method of claim 1 , wherein the linearized representation of the hierarchical parse tree for the input text segment is a sequence of symbols from a predetermined vocabulary of parse tree symbols.

4. The method of claim 1 , wherein the input text segment is a variable length input text segment.

5. The method of claim 1 , wherein the alternative representation is a vector of fixed dimensionality.

6. The method of claim 1 , wherein processing the input text segment comprises: adding an end-of-sentence token to the end of the input text segment to generate a modified input text segment; and processing the modified input text segment using the first LSTM neural network.

7. The method of claim 1 , wherein processing the alternative representation for the input text segment using the second LSTM neural network comprises initializing a hidden state of the second LSTM neural network to the alternative representation for the input text segment.

8. The method of claim 1 , wherein processing the alternative representation for the input text segment using the second LSTM neural network comprises: processing the alternative representation for the input text segment using the second LSTM neural network to generate a respective sequence score for each of a plurality of possible linearized representations of hierarchical parse trees; and selecting a possible linearized representation having a highest sequence score as the linearized representation of the hierarchical parse tree for the input text segment.

9. The method of claim 8 , wherein processing the alternative representation for the input text segment using the second LSTM neural network to generate a respective sequence score for each of a plurality of possible linearized representations of hierarchical parse trees comprises: processing the alternative representation using the second LSTM neural network using a left to right beam search decoding.

10. The method of claim 8 , wherein the set of possible linearized representations comprises possible linearized representations of varying lengths.

11. The method of claim 1 , further comprising: training the first LSTM neural network and the second LSTM neural network using Stochastic Gradient Descent.

12. The method of claim 1 , wherein one or more of the first LSTM neural network or the second LSTM neural network is a deep LSTM neural network.

13. The method of claim 1 , wherein the hierarchical parse tree is a tree that represents the syntactic structure of the text segment according to a context-free grammar, and wherein a linearized representation of a particular hierarchical parse tree is generated by traversing the particular hierarchical parse tree in a depth-first traversal order.

14. The method of claim 1 , comprising using the second LSTM neural network to sequentially select symbols for the linearized representation until an end-of-sentence token is selected that indicates the linearized representation is complete.

15. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: obtaining an input text segment; processing the input text segment using a first long short term memory (LSTM) neural network to convert the input text segment into an alternative representation for the input text segment; and processing the alternative representation for the input text segment using a second LSTM neural network to generate a linearized representation of a hierarchical parse tree for the input text segment, including using the second LSTM neural network to sequentially select symbols for the linearized representation, the selected symbols including (i) first symbols that represent syntactic elements of the input text segment and (ii) second symbols that represent hierarchical relationships between particular first symbols in the hierarchical parse tree.

16. The system of claim 15 , the operations further comprising: generating the hierarchical parse tree for the input text segment from the linearized representation of the hierarchical parse tree.

17. The system of claim 15 , wherein the linearized representation of the hierarchical parse tree for the input text segment is a sequence of symbols from a predetermined vocabulary of parse tree symbols.

18. The system of claim 15 , wherein processing the alternative representation for the input text segment using the second LSTM neural network comprises initializing a hidden state of the second LSTM neural network to the alternative representation for the input text segment.

19. A computer program product encoded on one or more non-transitory computer storage media, the computer program product comprising instruction that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining an input text segment; processing the input text segment using a first long short term memory (LSTM) neural network to convert the input text segment into an alternative representation for the input text segment; and processing the alternative representation for the input text segment using a second LSTM neural network to generate a linearized representation of a hierarchical parse tree for the input text segment, including using the second LSTM neural network to sequentially select symbols for the linearized representation, the selected symbols including (i) first symbols that represent syntactic elements of the input text segment and (ii) second symbols that represent hierarchical relationships between particular first symbols in the hierarchical parse tree.

Assignees

• Google Llc

Inventors

• Vinyals Oriol
• Kaiser Lukasz Mieczyslaw

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F40/205 Primary

  Parsing · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0454

  Physics · mapped topic

• G06F17/2705 Primary

  Physics · mapped topic