Language model for abstractive summarization

Patent metadata

Field	Value
Publication number	US-11475210-B2
Application number	US-202117304081-A
Country	US
Kind code	B2
Filing date	Jun 14, 2021
Priority date	Aug 31, 2020
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Methods, systems, and computer programs are presented for abstractive summarization of text by viewing sequence transduction as a language modeling problem. One method comprises an operation for training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for the text being summarized. The method further comprises operations for detecting the text to be summarized, initializing the running summary, and performing a plurality of iterations. Each iteration comprises providing, to the machine-learning model, the source text and the running summary, and adding, using the machine-learning model, a new word to the running summary. Further, the method comprises an operation for storing, on a memory, the running summary as the summary of the text.

First claim

The invention claimed is:

1. A computer-implemented method for creating a summary for text, the method comprising:
training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for the text being summarized;
detecting the text to be summarized;
initializing the running summary;
performing a plurality of iterations, each iteration comprising:
providing, to the machine-learning model, the text and the running summary; and
adding, using the machine-learning model, a new word to the running summary; and

storing, on a memory, the running summary as the summary of the text.

2. The method as recited in claim 1 , wherein the training is based on training data, the training data comprising a plurality of conversations and corresponding summaries.

3. The method as recited in claim 2 , wherein the machine-learning program is trained using maximum likelihood, wherein the training data comprises, for each conversation from the plurality of conversations, the conversation, a control token, and the summary of the conversation, the control token separating the conversation from the summary.

4. The method as recited in claim 1 , wherein the machine-learning program is a decoder-only deep-learning transformer.

5. The method as recited in claim 4 , wherein the decoder-only deep-learning transformer comprises four layers comprising: a masked self attention layer, a first norm layer, a feed forward layer, and a second norm layer.

6. The method as recited in claim 1 , wherein the text is embedded using data-driven subword encoding via Byte Pair Encoding (BPE).

7. The method as recited in claim 1 , wherein initializing the running summary comprises setting the running summary to be empty.

8. The method as recited in claim 1 , wherein the text is a conversation that comprises one or more turns.

9. The method as recited in claim 1 , wherein the text is a turn within a conversation.

10. The method as recited in claim 1 , further comprising:
causing presentation of the summary on a display.

11. A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations comprising:
training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for text being summarized;
detecting the text to be summarized;
initializing the running summary;
performing a plurality of iterations, each iteration comprising:
providing, to the machine-learning model, the text and the running summary; and
adding, using the machine-learning model, a new word to the running summary; and

storing, on the memory, the running summary as a summary of the text.

12. The system as recited in claim 11 , wherein the training is based on training data, the training data comprising a plurality of conversations and corresponding summaries.

13. The system as recited in claim 12 , wherein the machine-learning program is trained using maximum likelihood, wherein the training data comprises, for each conversation from the plurality of conversations, the conversation, a control token, and the summary of the conversation, the control token separating the conversation from the summary.

14. The system as recited in claim 11 , wherein the machine-learning program is a decoder-only deep-learning transformer.

15. The system as recited in claim 14 , wherein the decoder-only deep-learning transformer comprises four layers comprising: a masked self attention layer, a first norm layer, a feed forward layer, and a second norm layer.

16. A tangible machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for text being summarized;
detecting the text to be summarized;
initializing the running summary;
performing a plurality of iterations, each iteration comprising:
providing, to the machine-learning model, the text and the running summary; and
adding, by the machine-learning model, a new word to the running summary; and

storing, on a memory, the running summary as a summary of the text.

17. The tangible machine-readable storage medium as recited in claim 16 , wherein the training is based on training data, the training data comprising a plurality of conversations and corresponding summaries.

18. The tangible machine-readable storage medium as recited in claim 17 , wherein the machine-learning program is trained using maximum likelihood, wherein the training data comprises, for each conversation from the plurality of conversations, the conversation, a control token, and the summary of the conversation, the control token separating the conversation from the summary.

19. The tangible machine-readable storage medium as recited in claim 16 , wherein the machine-learning program is a decoder-only deep-learning transformer.

20. The tangible machine-readable storage medium as recited in claim 19 , wherein the decoder-only deep-learning transformer comprises four layers comprising: a masked self attention layer, a first norm layer, a feed forward layer, and a second norm layer.

Assignees

• Twilio Inc

Inventors

• De Oliveira Luke Percival
• Láinez Rodrigo Alfredo

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F40/284

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06F40/166 Primary

  Editing, e.g. inserting or deleting · CPC title

• G06F40/35 Primary

  Discourse or dialogue representation · CPC title

• G06N3/08

  Learning methods · CPC title