Implementing text generation

Patent metadata

Field	Value
Publication number	US-11675983-B2
Application number	US-202117331526-A
Country	US
Kind code	B2
Filing date	May 26, 2021
Priority date	Dec 22, 2020
Publication date	Jun 13, 2023
Grant date	Jun 13, 2023

Abstract

A method for implementing text generation, a device and a medium are provided. The method includes: determining a target task type of a target text generation task from multiple task types supported by a pre-trained general text generation model; determining, based on a requirement of the target text generation task for a target output text, a first target output text attribute for the target text generation task from multiple output text attributes supported by the general text generation model; and fine tuning the general text generation model based on a target training data set associated with the target text generation task to obtain a task-specific text generation model, by taking task indication information for the target task type and first attribute indication information for the first target output text attribute as at least part of an input of the general text generation model.

First claim

The invention claimed is:

1. A method, comprising:
determining a target task type of a target text generation task from multiple task types supported by a general text generation model, wherein the general text generation model is pre-trained;
determining, based on a requirement of the target text generation task for a target output text, at least one first target output text attribute for the target text generation task from multiple output text attributes supported by the general text generation model;
determining second attribute indication information indicating a second target output text attribute for the target text generation task; and
fine tuning the general text generation model based on a target training data set associated with the target text generation task to obtain a task-specific text generation model for the target text generation task, by:
inputting task indication information for the target task type and first attribute indication information for the at least one first target output text attribute to a first encoder in the general text generation model;
inputting the second attribute indication information to a second encoder in the general text generation model; and
generating an output text based on an intermediate output of the first encoder and an intermediate output of the second encoder,

wherein the target training data set is an additional data set other than pre-training data sets for pre-training the general text generation model.

2. The method according to claim 1 , wherein the general text generation model is pre-trained based on multiple pre-training data sets respectively comprising training input texts and training output texts associated with corresponding task types in the multiple task types.

3. The method according to claim 2 , wherein a training output text in at least one pre-training data set of the multiple pre-training data sets is labeled with different attribute values of at least one output text attribute of the multiple output text attributes.

4. The method according to claim 1 , wherein fine tuning the general text generation model further comprises:
obtaining an attribute control model configured to control the decoder to output the target output text having a third target output text attribute based on the intermediate output of the first encoder and the intermediate output of the second encoder, wherein the third target output text attribute is not included in the multiple output text attributes; and
fine tuning the attribute control model and the general text generation model jointly.

5. The method according to claim 4 , wherein the attribute control model comprises at least one of:
an attribute classification model configured to determine, based on the intermediate output of the first encoder and the intermediate output of the second encoder, guidance information for guiding the decoder to generate the target output text having the third target output text attribute, wherein the guidance information indicates different attribute values of the third target output text attribute; and
a language model configured to, in conjunction with the decoder, decode the intermediate output of the first encoder and the intermediate output of the second encoder to generate the target output text having the third target output text attribute.

6. The method according to claim 1 , wherein the target text generation task is a first target text generation task, and the method further comprises:
determining a second target task type of a second target text generation task from the multiple task types;
determining, based on a requirement of the second target text generation task for a target output text, a further target output text attribute for the second target text generation task from the multiple output text attributes; and
fine tuning the general text generation model based on a second target training data set associated with the second target text generation task to obtain a further task-specific text generation model for the second target text generation task, by taking second task indication information for the second target task type and further attribute indication information for the further target output text attribute as at least part of an input of the general text generation model.

7. The method according to claim 1 , further comprising:
obtaining a target input text for the target text generation task;
applying the target input text, the task indication information and the first attribute indication information as an input of the task-specific text generation model; and
executing the task-specific text generation model to obtain an output of the task-specific text generation model as the target output text of the target text generation task.

8. An electronic device, comprising:
at least one processor; and
a memory communicatively connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to:
determine a target task type of a target text generation task from multiple task types supported by a general text generation model, wherein the general text generation model is pre-trained;
determine at least one first target output text attribute for the target text generation task from multiple output text attributes supported by the general text generation model based on a requirement of the target text generation task for a target output text;
determine second attribute indication information indicating a second target output text attribute for the target text generation task; and
fine tune the general text generation model based on a target training data set associated with the target text generation task to obtain a task-specific text generation model for the target text generation task, by:
inputting task indication information for the target task type and first attribute indication information for the at least one first target output text attribute to a first encoder in the general text generation model;
inputting the second attribute indication information to a second encoder in the general text generation model; and
generating an output text based on an intermediate output of the first encoder and an intermediate output of the second encoder,

wherein the target training data set is an additional data set other than pre-training data sets for pre-training the general text generation model.

9. The electronic device according to claim 8 , wherein the general text generation model is pre-trained based on multiple pre-training data sets respectively comprising training input texts and training output texts associated with corresponding task types in the multiple task types.

10. The electronic device according to claim 9 , wherein a training output text in at least one pre-training data set of the multiple pre-training data sets is labeled with different attribute values of at least one output text attribute of the multiple output text attributes.

11. The electronic device according to claim 8 , wherein fine tuning the general text generation model comprises:
obtaining an attribute control model configured to control the decoder to output the target output text having a third target output text attribute based on the intermediate output of the first encoder and the intermediate output of the second encoder, wherein the third target output text attribute is not included in the multiple output text attributes; and
fine tuning the attribute control model and the general text generation model jointly.

12. The electronic device according to claim 11 , wherein the attribute control model comprises at least on

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Liu Jiachen
• Hu Zhe
• Xiao Xinyan
• Wu Hua

Classifications

• G06F40/117

  Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154) · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/56 Primary

  Natural language generation · CPC title

• G06F40/126

  Character encoding · CPC title

• G06N20/00

  Machine learning · CPC title