Systems and methods for unifying question answering and text classification via span extraction

Patent metadata

Field	Value
Publication number	US-2020334334-A1
Application number	US-201916518905-A
Country	US
Kind code	A1
Filing date	Jul 22, 2019
Priority date	Apr 18, 2019
Publication date	Oct 22, 2020
Grant date	—

Abstract

Systems and methods for unifying question answering and text classification via span extraction include a preprocessor for preparing a source text and an auxiliary text based on a task type of a natural language processing task, an encoder for receiving the source text and the auxiliary text from the preprocessor and generating an encoded representation of a combination of the source text and the auxiliary text, and a span-extractive decoder for receiving the encoded representation and identifying a span of text within the source text that is a result of the NLP task. The task type is one of entailment, classification, or regression. In some embodiments, the source text includes one or more of text received as input when the task type is entailment, a list of classifications when the task type is entailment or classification, or a list of similarity options when the task type is regression.

First claim

What is claimed is:

1 . A system for performing a natural language processing (NLP) task comprising:
a preprocessor for preparing a source text and an auxiliary text based on a task type of the NLP task, the task type being entailment, classification, or regression; an encoder for receiving the source text and the auxiliary text from the preprocessor and generating an encoded representation of a combination of the source text and the auxiliary text; and a span-extractive decoder for receiving the encoded representation and identifying a span of text within the source text that is a result of the NLP task.

2 . The system of claim 1 , wherein the encoder is a multi-layer attention-based encoder.

3 . The system of claim 1 , wherein the span-extractive decoder comprises:
a first softmax for combining a trainable parameter vector associated with start token positions of the span of text and a portion of the encoded representation corresponding to the source text and generating a distribution of possible start tokens for the span of text; a first argument maximum module for selecting a start token for the span of text based on the distribution of possible start tokens for the span of text; a second softmax for combining a trainable parameter vector associated with end token positions of the span of text and the portion of the encoded representation corresponding to the source text and generating a distribution of possible end tokens for the span of text; and a second argument maximum module for selecting an end token for the span of text based on the distribution of possible end tokens for the span of text.

4 . The system of claim 1 , wherein the preprocessor further:
receives one or more text inputs; and uses one of the one or more text inputs as the auxiliary text.

5 . The system of claim 4 , wherein the preprocessor further uses another of the one or more text inputs as part of the source text when the task type is entailment or regression.

6 . The system of claim 1 , wherein the preprocessor further includes a list of classifications in the source text when the task type is entailment or classification.

7 . The system of claim 6 , wherein the list of classifications is included in one of the one or more text inputs.

8 . The system of claim 6 , wherein the list of classifications is looked-up based on the task type.

9 . The system of claim 1 , wherein the preprocessor further includes a list of similarity options in the source text when the task type is regression.

10 . The system of claim 1 , wherein the preprocessor further generates an embedding for the combination of the start text and the auxiliary text, the embedding including information as to whether an embedded token corresponds to a token in the start text or a token in the auxiliary text.

11 . A method for performing an natural language processing (NLP) task comprising:
preparing, using a preprocessing module, a source text and an auxiliary text based on a task type of the NLP task, the task type being entailment, classification, or regression; generating, using an encoding module, an encoded representation of a combination of the source text and the auxiliary text; and identifying, using a span-extractive decoding module, a span of text within the source text that is a result of the NLP task.

12 . The method of claim 11 , wherein generating the encoded representation comprises using a plurality of attention-based encoding layers.

13 . The method of claim 11 , wherein identifying the span of text comprises:
combining a trainable parameter vector associated with start token positions of the span of text and a portion of the encoded representation corresponding to the source text to generate a distribution of possible start tokens for the span of text; selecting a start token for the span of text based on the distribution of possible start tokens for the span of text; combining a trainable parameter vector associated with end token positions of the span of text and the portion of the encoded representation corresponding to the source text and generating a distribution of possible end tokens for the span of text; and selecting an end token for the span of text based on the distribution of possible end tokens for the span of text.

14 . The method of claim 11 , wherein preparing the auxiliary text comprises receiving the auxiliary text as an input.

15 . The method of claim 11 , wherein preparing the source text comprises receiving a portion of the source text as an input when the task type is entailment or regression.

16 . The method of claim 11 , wherein preparing the source text comprises including a list of classifications in the source text when the task type is entailment or classification, the list of classifications being received as an input or being looked-up based on the task type.

17 . The method of claim 11 , wherein preparing the source text comprises including a list of similarity options in the source text when the task type is regression.

18 . The method of claim 11 , further comprising generating an embedding for the combination of the start text and the auxiliary text, the embedding including information as to whether an embedded token corresponds to a token in the start text or a token in the auxiliary text.

19 . A non-transitory machine-readable medium comprising executable code which when executed by one or more processors associated with a computing device are adapted to cause the one or more processors to perform a method comprising:
preparing a first text string and a second text string based on a task type of an NLP task, the task type being entailment, classification, or regression; generating an encoded representation of a combination of the first text string and the second text string; identifying a span of text within the first text string that is a result of the NLP task by:
combining a trainable parameter vector associated with start token positions of the span of text and a portion of the encoded representation corresponding to the first text string to generate a distribution of possible start tokens for the span of text;
selecting a start token for the span of text based on the distribution of possible start tokens for the span of text;
combining a trainable parameter vector associated with end token positions of the span of text and the portion of the encoded representation corresponding to the first text string and generating a distribution of possible end tokens for the span of text; and
selecting an end token for the span of text based on the distribution of possible end tokens for the span of text.

20 . The non-transitory machine-readable medium of claim 19 , wherein preparing the first text string comprises one or more of:
receiving a portion of the first text string as an input when the task type is entailment or regression; including a list of classifications in the first text string when the task type is entailment or classification, the list of classifications being received as an input or being looked-up based on the task type; or including a list of similarity options in the first text string when the task type is regression.

Assignees

• Salesforce Com Inc

Inventors

• Keskar Nitish Shirish
• Mccann Bryan
• Socher Richard
• Xiong Caiming

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06F40/284

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06N3/08

  Learning methods · CPC title

• G06F16/3329

  Natural language query formulation · CPC title