System and method for natural language processing using neural network with cross-task training

Patent metadata

Field	Value
Publication number	US-12086539-B2
Application number	US-202017093478-A
Country	US
Kind code	B2
Filing date	Nov 9, 2020
Priority date	Dec 9, 2019
Publication date	Sep 10, 2024
Grant date	Sep 10, 2024

Abstract

A method for using a neural network model for natural language processing (NLP) includes receiving training data associated with a source domain and a target domain; and generating one or more query batches. Each query batch includes one or more source tasks associated with the source domain and one or more target tasks associated with the target domain. For each query batch, class representations are generated for each class in the source domain and the target domain. A query batch loss for the query batch is generated based on the corresponding class representations. An optimization is performed on the neural network model by adjusting its network parameters based on the query batch loss. The optimized neural network model is used to perform one or more new NLP tasks.

First claim

What is claimed is:

1. A method for using a neural network model for natural language processing (NLP), comprising:
receiving training data associated with a source domain and a target domain;
generating one or more query batches,
wherein a source subquery batch includes one or more source tasks associated with the source domain, and
wherein a target subquery batch includes one or more target tasks associated with the target domain;
for each query batch, generating combination class representations associated with a combination of classes in the source domain and the target domain;
generating a source loss using the source subquery batch and the combination class representations;
generating a target loss using the target subquery batch and the combination class representations; and
generating a query batch loss using the source loss and target loss; and
performing an optimization on the neural network model by adjusting its network parameters based on the query batch loss,
wherein the optimized neural network model is used to perform one or more new NLP tasks.

2. The method of claim 1 , wherein a first new NLP task is from one of the target domain and a new domain, wherein the new domain is different from the source domain and target domain.

3. The method of claim 2 , wherein a second new NLP task is from the other of the target domain and the new domain.

4. The method of claim 1 , wherein the neural network model includes a textual entailment model,
wherein the one or more source tasks and the one or more target tasks include one or more textual entailment tasks regarding a relation of a premise sentence including a premise and a hypothesis sentence including a hypothesis, where the relation indicates whether the hypothesis is true given the premise.

5. The method of claim 1 , wherein the generating the source loss includes:
generating a probability distribution by comparing a query of the source subquery batch with the combination class representations; and
generating the source loss based on the probability distribution.

6. The method of claim 1 , wherein the generating the target loss includes:
generating a probability distribution by comparing a query of the target subquery batch with the combination class representations; and
generating the target loss based on the probability distribution.

7. The method of claim 1 , wherein few-shot learning is performed to the neural network model.

8. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising:
receiving training data associated with a source domain and a target domain;
generating one or more query batches,
wherein a source subquery batch includes one or more source tasks associated with the source domain, and
wherein a target subquery batch includes one or more target tasks associated with the target domain;

for each query batch, generating combination class representations associated with a combination of classes in the source domain and the target domain;
generating a source loss using the source subquery batch and the combination class representations;
generating a target loss using the target subquery batch and the combination class representations; and
generating a query batch loss using the source loss and target loss; and
performing an optimization on the neural network model by adjusting its network parameters based on the query batch loss,
wherein the optimized neural network model is used to perform one or more new NLP tasks.

9. The non-transitory machine-readable medium of claim 8 , wherein a first new NLP task is from one of the target domain and a new domain different from the source domain and target domain.

10. The non-transitory machine-readable medium of claim 9 , wherein a second new NLP task is from the other of the target domain and the new domain.

11. The non-transitory machine-readable medium of claim 8 , wherein the neural network model includes a textual entailment model,
wherein the one or more source tasks and one or more target tasks include one or more textual entailment tasks regarding a relation of a premise sentence including a premise and a hypothesis sentence including a hypothesis, where the relation indicates whether the hypothesis is true given the premise.

12. The non-transitory machine-readable medium of claim 8 , wherein the generating the source loss includes:
generating a probability distribution by comparing a query of the source subquery batch with the combination class representations; and
generating the source loss based on the probability distribution.

13. The non-transitory machine-readable medium of claim 8 , wherein the generating the target loss includes:
generating a probability distribution by comparing a query of the target subquery batch with the combination class representations; and
generating the target loss based on the probability distribution.

14. The non-transitory machine-readable medium of claim 12 , wherein the generating the class representations for each query batch includes:
generating source sample sets for the classes from the source domain; and
generating the class representations based on the source sample sets and the target subquery batch.

15. A system, comprising:
a non-transitory memory; and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform a method comprising:
receiving training data associated with a source domain and a target domain;
generating one or more query batches,
wherein a source subquery batch includes one or more source tasks associated with the source domain, and
wherein a target subquery batch includes one or more target tasks associated with the target domain;

for each query batch, generating combination class representations associated with a combination of classes in the source domain and the target domain;
generating a source loss using the source subquery batch and the combination class representations;
generating a target loss using the target subquery batch and the combination class representations; and
generating a query batch loss using the source loss and target loss; and
performing an optimization on the neural network model by adjusting its network parameters based on the query batch loss,
wherein the optimized neural network model is used to perform one or more new NLP tasks.

16. The system of claim 15 , wherein a first new NLP task is associated with one of the target domain and a new domain, wherein the new domain is different from the source domain and target domain.

17. The system of claim 16 , wherein a second new NLP task is associated with the other of the target domain and the new domain.

18. The system of claim 15 , wherein the neural network model includes a textual entailment model,
wherein the one or more source tasks and one or more target tasks include one or more textual entailment tasks regarding a relation of a premise sentence including a premise and a hypothesis sentence including a hypothesis, where the relation indicates whether the hypothesis is true given the premise.

19. The system of claim 15 , wherein the generating the source loss includes:
generating a probability distribution by comparing a query of the source subquery batch with the combination class representations; and
generating the source loss

Assignees

• Salesforce Inc

Inventors

• Yin Wenpeng
• Rajani Nazneen
• Socher Richard
• Xiong Caiming

Classifications

• G06N3/096

  Transfer learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06F16/3344

  using natural language analysis · CPC title

• G06F40/279

  Recognition of textual entities · CPC title