Salience-aware cross-attention for abstractive summarization

Patent metadata

Field	Value
Publication number	US-2024193973-A1
Application number	US-202218078155-A
Country	US
Kind code	A1
Filing date	Dec 9, 2022
Priority date	Dec 9, 2022
Publication date	Jun 13, 2024
Grant date	—

Abstract

A method including: receiving an input comprising natural language texts at an encoder; adding a token to the input; obtaining a last-layer hidden state as a natural language text representation; feeding the natural language text representation into a single-layer classification head; predicting a salience allocation based on the single-layer classification head; developing a salience-aware cross-attention (SACA) decoder to determine salience in the natural language text representation; mapping a plurality of salience degrees to a plurality of trainable salience embeddings; estimating an amount of signal to accept from the plurality of trainable salience embeddings; incorporating the salience allocation and the signal in a cross-attention layer model; and generating a summarization based on the SACA decoder and the cross-attention layer model.

First claim

What is claimed is:

1 . A method executed by at least one processor, the method comprising:
receiving an input comprising natural language texts at an encoder; adding a token to the input; obtaining a last-layer hidden state as a natural language text representation; feeding the natural language text representation into a single-layer classification head; predicting a salience allocation based on the single-layer classification head; developing a salience-aware cross-attention (SACA) decoder to determine salience in the natural language text representation; mapping a plurality of salience degrees to a plurality of trainable salience embeddings; estimating an amount of signal to accept from the plurality of trainable salience embeddings; incorporating the salience allocation and the signal in a cross-attention layer model; and generating a summarization based on the SACA decoder and the cross-attention layer model.

2 . The method according to claim 1 , wherein the natural language text representation comprises a contextualized embedding of the natural language texts and a modified input sequence of the natural language texts.

3 . The method according to claim 1 , further comprising assigning a ground-truth salience label to the natural language texts based on a similarity between the natural language texts and a ground-truth summary.

4 . The method according to claim 1 , wherein estimating the amount of the signal to accept from the plurality of trainable salience embeddings comprises identifying a salience embedding of a salience degree that maximizes a probability.

5 . The method according to claim 1 , wherein the plurality of salience degrees have a corresponding plurality of cut-off thresholds that are based on a corpus to balance informativeness and prediction accuracy.

6 . The method according to claim 1 , wherein predicting the salience allocation and generating the summarization occurs simultaneously.

7 . The method according to claim 1 , wherein predicting the salience allocation further comprises:
averaging a cross-entropy loss in each natural language text of the natural language texts; applying label smoothing to the plurality of salience degrees for denoising; and assigning a probability to the plurality of salience degrees adjacent to a ground-truth.

8 . An apparatus comprising:
at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
receiving code configured to cause the at least one processor to receive an input comprising natural language texts at an encoder;
adding code configured to cause the at least one processor to add a token to the input;
obtaining code configured to cause the at least one processor to obtain a last-layer hidden state as a natural language text representation;
feeding code configured to cause the at least one processor to feed the natural language text representation into a single-layer classification head;
predicting code configured to cause the at least one processor to predict a salience allocation based on the single-layer classification head;
developing code configured to cause the at least one processor to develop a salience-aware cross-attention (SACA) decoder to determine salience in the natural language text representation;
mapping code configured to cause the at least one processor to map a plurality of salience degrees to a plurality of trainable salience embeddings;
estimating code configured to cause the at least one processor to estimate an amount of signal to accept from the plurality of trainable salience embeddings;
incorporating code configured to cause the at least one processor to incorporate the salience allocation and the signal in a cross-attention layer model; and
generating code configured to cause the at least one processor to generate a summarization based on the SACA decoder and the cross-attention layer model.

9 . The apparatus according to claim 8 , wherein the natural language text representation comprises a contextualized embedding of the natural language texts and a modified input sequence of the natural language texts.

10 . The apparatus according to claim 8 , wherein the program code further comprises assigning code configured to cause the at least one processor to assign a ground-truth salience label to the natural language texts based on a similarity between the natural language texts and a ground-truth summary.

11 . The apparatus according to claim 8 , wherein the estimating code further causes the at least one processor to identify a salience embedding of a salience degree that maximizes a probability.

12 . The apparatus according to claim 8 , wherein the plurality of salience degrees have a corresponding plurality of cut-off thresholds that are based on a corpus to balance informativeness and prediction accuracy.

13 . The apparatus according to claim 8 , wherein predicting the salience allocation and generating the summarization occurs simultaneously.

14 . The apparatus according to claim 8 , wherein the predicting code further causes the at least one processor to:
average a cross-entropy loss in each natural language text of the natural language texts; apply label smoothing to the plurality of salience degrees for denoising; and assign a probability to the plurality of salience degrees adjacent to a ground-truth.

15 . A non-transitory computer-readable storage medium, storing instructions, which, when executed by at least one processor, cause the at least one processor to:
receive an input comprising natural language texts at an encoder; add a token to the input; obtain a last-layer hidden state as a natural language text representation; feed the natural language text representation into a single-layer classification head; predict a salience allocation based on the single-layer classification head; develop a salience-aware cross-attention (SACA) decoder to determine salience in the natural language text representation; map a plurality of salience degrees to a plurality of trainable salience embeddings; estimate an amount of signal to accept from the plurality of trainable salience embeddings; incorporate the salience allocation and the signal in a cross-attention layer model; and generate a summarization based on the SACA decoder and the cross-attention layer model.

16 . The non-transitory computer-readable storage medium according to claim 15 , wherein the natural language text representation comprises a contextualized embedding of the natural language texts and a modified input sequence of the natural language texts.

17 . The non-transitory computer-readable storage medium according to claim 15 , wherein the instructions further cause the at least one processor to assign a ground-truth salience label to the natural language texts based on a similarity between the natural language texts and a ground-truth summary.

18 . The non-transitory computer-readable storage medium according to claim 15 , wherein the instructions that cause the at least one processor to estimate the amount of the signal to accept from the plurality of trainable salience embeddings further causes the at least one processor to identify a salience embedding of a salience degree that maximizes a probability.

19 . The non-transitory computer-readable storage medium according to claim 15 , wherein the plurality of salience degrees have a corresponding plurality of cut

Assignees

• Tencent America LLC

Inventors

• Song Kaiqiang
• Wang Fei
• Wang Xiaoyang
• Cho Sangwoo
• Yu Dong

Classifications

• G06F40/30

  Semantic analysis · CPC title

• G06V30/164

  Noise filtering · CPC title

• G06V30/19093

  Proximity measures, i.e. similarity or distance measures · CPC title

• G06F40/284

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06V30/18152 Primary

  Extracting features based on a plurality of salient regional features, e.g. "bag of words" · CPC title