Systems and methods for question-driven pretraining for controllable summarization

Patent metadata

Field	Value
Publication number	US-2024202461-A1
Application number	US-202318299503-A
Country	US
Kind code	A1
Filing date	Apr 12, 2023
Priority date	Dec 15, 2022
Publication date	Jun 20, 2024
Grant date	—

Abstract

Embodiments described herein provide a task-targeted training paradigm which involves reformulating the pretraining objective to match the downstream task, e.g., query-focused summarization with or without guidance, without additional supervision. Specifically, query-focused summarization is foremost a task in which information is selected and compressed. A training objective is computed based on asking questions about sentences considered to be informative about the text that naturally incorporates a form of guidance of the generation process.

First claim

What is claimed is:

1 . A method of question-driven summarization, the method comprising:
receiving, via a data interface, a training dataset containing at least one unlabeled document; selecting, by a selection model implemented on one or more processors, a plurality of sentences from the unlabeled document depending on an information overlap metric with a rest of the unlabeled document; generating, by a question generation model implemented on the one or more processors, a plurality of questions in response to an input of the plurality of sentences and the unlabeled document as context; generating a masked document by masking the plurality of sentences from the unlabeled document with mask tokens; generating, by a summarization model implemented on the one or more processors, predicted questions in response to an input of the masked document; computing a first loss comparing the predicted questions and the plurality of questions; and training the summarization model by updating parameters of the summarization model based on the computed first loss via backpropagation.

2 . The method of claim 1 , wherein the plurality of sentences are selected by a metric that measure information overlap of each of the plurality of sentences with a rest of the input document.

3 . The method of claim 1 , further comprising:
generating, by the summarization model, a predicted summary containing one or more sentences in response to an input of the masked document prepended with the plurality of questions; computing a second loss comparing the predicted one or more sentences and the plurality of sentences; and training the summarization model by updating parameters of the summarization model based on the computed second loss via backpropagation.

4 . The method of claim 1 , further comprising:
generating, by a summarization model implemented on the one or more processors, predicted questions in response to an input of the masked document; generating, by the summarization model, a predicted summary containing one or more sentences conditioned on the predicted questions and the masked document; computing a second loss comparing the predicted questions and the plurality of questions and comparing the one or more predicted sentences and the plurality of sentences; and training the summarization model by updating parameters of the summarization model based on the computed second loss via backpropagation.

5 . The method of claim 1 , further comprising:
generating, by the summarization model, reconstructed sentences in response to an input of the masked document; computing a second loss comparing the reconstructed sentences and the plurality of sentences; and training the summarization model by updating parameters of the summarization model based on the computed second loss via backpropagation.

6 . The method of claim 1 , further comprising:
prepending a dedicated token to the masked document before inputting the masked document to the summarization model; and determining a type of training output depending on the dedicated token.

7 . The method of claim 1 , further comprising:
receiving a testing document and a user query relating to a content of the testing document; and generating, by the trained summarization model, a question-driven summary of the testing document according to the user query.

8 . A system of question-driven summarization, the system comprising:
a communication interface that receives a training dataset containing at least one unlabeled document; a memory storing a selection model, a question generation model, a summarization model and a plurality of processor-executable instructions; and one or more processors executing the plurality of processor-executable instructions to perform operations comprising:
selecting, by the selection model, a plurality of sentences from the unlabeled document depending on an information overlap metric with a rest of the unlabeled document;
generating, by the question generation model, a plurality of questions in response to an input of the plurality of sentences and the unlabeled document as context;
generating the masked document by masking the plurality of sentences from the unlabeled document with mask tokens;
generating, by the summarization model, a predicted summary containing one or more sentences in response to an input of the masked document prepended with the plurality of questions;
computing a first loss comparing the predicted one or more sentences and the plurality of sentences; and
training the summarization model by updating parameters of the summarization model based on the computed first loss via backpropagation.

9 . The system of claim 8 , wherein the plurality of sentences are selected by a metric that measure information overlap of each of the plurality of sentences with a rest of the input document.

10 . The system of claim 8 , wherein the operations further comprise:
generating, by the summarization model implemented on the one or more processors, predicted questions in response to an input of the masked document; computing a second loss comparing the predicted questions and the plurality of questions; and training the summarization model by updating parameters of the summarization model based on the computed second loss via backpropagation.

11 . The system of claim 8 , wherein the operations further comprise:
generating, by a summarization model implemented on the one or more processors, predicted questions in response to an input of the masked document; generating, by the summarization model, a predicted summary containing one or more sentences conditioned on the predicted questions and the masked document; computing a second loss comparing the predicted questions and the plurality of questions and comparing the one or more predicted sentences and the plurality of sentences; and training the summarization model by updating parameters of the summarization model based on the computed second loss via backpropagation.

12 . The system of claim 8 , wherein the operations further comprise:
generating, by the summarization model, reconstructed sentences in response to an input of the masked document; computing a second loss comparing the reconstructed sentences and the plurality of sentences; and training the summarization model by updating parameters of the summarization model based on the computed second loss via backpropagation.

13 . The system of claim 8 , wherein the operations further comprise:
prepending a dedicated token to the masked document before inputting the masked document to the summarization model; and determining a type of training output depending on the dedicated token.

14 . The system of claim 8 , wherein the operations further comprise:
receiving a testing document and a user query relating to a content of the testing document; and generating, by the trained summarization model, a question-driven summary of the testing document according to the user query.

15 . A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for question-driven summarization, the instructions being executed by one or more processors to perform operations comprising:
receiving, via a data interface, a training dataset containing at least one unlabeled document; selecting, by a selection model implemented on one or more processors, a plurality of sentences from the unlabeled document depending on an information overlap metric with a rest of the unlabeled document; generating, by a question generation model implemented on the one

Assignees

• Salesforce Inc

Inventors

• Pagnoni Artidoro
• Fabbri Alexander
• Kryscinski Wojciech

Classifications

• G06F16/345

  Summarisation for human users · CPC title

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title