Method and apparatus for machine reading comprehension, and non-transitory computer-readable recording medium

Patent metadata

Field	Value
Publication number	US-12175206-B2
Application number	US-202217821227-A
Country	US
Kind code	B2
Filing date	Aug 22, 2022
Priority date	Sep 7, 2021
Publication date	Dec 24, 2024
Grant date	Dec 24, 2024

Abstract

A method and an apparatus for machine reading comprehension, and a non-transitory computer-readable recording medium are provided. In the method, a paragraph-question pair is obtained, and subword vectors corresponding to subwords in the paragraph-question pair are generated. Then, for each subword, relative positions of the subword with respect to the other subwords are determined based on distances, and self-attention information of the subword in a first part and mutual attention information of the subword in a second part are calculated by using the relative positions and the subword vector. Then, a fusion vector of the subword is generated based on the self-attention information and the mutual attention information. Then, the fusion vectors of the subwords are input to a decoder of a machine reading comprehension model so as to obtain an answer predicted by the decoder.

First claim

What is claimed is:

1. A method for machine reading comprehension, the method comprising:
obtaining a paragraph-question pair including a paragraph and a question, and generating subword vectors corresponding to subwords in the paragraph-question pair;
for each subword in the paragraph-question pair, determining, based on distances between the subword and other subwords in the paragraph-question pair, relative positions of the subword with respect to the other subwords in the paragraph-question pair, and calculating self-attention information of the subword in a first part and mutual attention information of the subword in a second part by using the relative positions and the subword vector of the subword, the first part being the paragraph or the question where the subword is located, in a case where the first part is the paragraph, the second part being the question, and in a case where the first part is the question, the second part being the paragraph;
for each subword in the paragraph-question pair, generating, based on the self-attention information and the mutual attention information of the subword, a fusion vector of the subword; and
inputting the fusion vectors of the subwords in the paragraph-question pair to a decoder of a machine reading comprehension model so as to obtain an answer predicted by the decoder.

2. The method for machine reading comprehension as claimed in claim 1 ,
wherein calculating the self-attention information of the subword in a first part includes
for each subword in the paragraph-question pair, calculating self-attention weights of subwords in the first part by using the relative positions of the subword with respect to the subwords in the first part; and
performing weighted summation on the subword vectors corresponding to the subwords in the first part by using the self-attention weights so as to obtain the self-attention information of the subword in the first part.

3. The method for machine reading comprehension as claimed in claim 1 ,
wherein calculating the mutual attention information of the subword in a second part includes
for each subword in the paragraph-question pair, calculating mutual attention weights of subwords in the second part by using the relative positions of the subword with respect to the subwords in the second part; and
performing weighted summation on the subword vectors corresponding to the subwords in the second part by using the mutual attention weights so as to obtain the mutual attention information of the subword in the second part.

4. The method for machine reading comprehension as claimed in claim 1 ,
wherein generating the fusion vector of the subword includes
for each subword in the paragraph-question pair, fusing the self-attention information and the mutual attention information of the subword so as to obtain the fusion vector of the subword.

5. The method for machine reading comprehension as claimed in claim 1 ,
wherein generating the fusion vector of the subword includes
for each subword in the paragraph-question pair, fusing the self-attention information and the mutual attention information of the subword so as to obtain attention information of the subword; and
fusing the attention information of the subword and word boundary information of the subword in a word to which the subword belongs so as to obtain the fusion vector of the subword.

6. The method for machine reading comprehension as claimed in claim 5 ,
wherein fusing the attention information and the word boundary information includes
determining, based on whether the subword is the first subword in the word to which the subword belongs, word starting information corresponding to the subword;
determining, based on whether the subword is the last subword in the word to which the subword belongs, word ending information corresponding to the subword; and
performing vector fusion on the attention information, the word starting information and the word ending information of the subword so as to obtain the fusion vector of the subword.

7. The method for machine reading comprehension as claimed in claim 4 ,
wherein fusing the self-attention information and the mutual attention information of the subword includes
performing vector addition or vector merging on the self-attention information and the mutual attention information of the subword.

8. An apparatus for machine reading comprehension, the apparatus comprising:
a memory storing computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions such that the one or more processors are configured to
obtain a paragraph-question pair including a paragraph and a question, and generate subword vectors corresponding to subwords in the paragraph-question pair;
for each subword in the paragraph-question pair, determine, based on distances between the subword and other subwords in the paragraph-question pair, relative positions of the subword with respect to the other subwords in the paragraph-question pair, and calculate self-attention information of the subword in a first part and mutual attention information of the subword in a second part by using the relative positions and the subword vector of the subword, the first part being the paragraph or the question where the subword is located, in a case where the first part is the paragraph, the second part being the question, and in a case where the first part is the question, the second part being the paragraph;
for each subword in the paragraph-question pair, generate, based on the self-attention information and the mutual attention information of the subword, a fusion vector of the subword; and
input the fusion vectors of the subwords in the paragraph-question pair to a decoder of a machine reading comprehension model so as to obtain an answer predicted by the decoder.

9. The apparatus for machine reading comprehension as claimed in claim 8 ,
wherein the one or more processors are configured to
for each subword in the paragraph-question pair, calculate self-attention weights of subwords in the first part by using the relative positions of the subword with respect to the subwords in the first part; and
perform weighted summation on the subword vectors corresponding to the subwords in the first part by using the self-attention weights so as to obtain the self-attention information of the subword in the first part.

10. The apparatus for machine reading comprehension as claimed in claim 8 ,
wherein the one or more processors are configured to
for each subword in the paragraph-question pair, calculate mutual attention weights of subwords in the second part by using the relative positions of the subword with respect to the subwords in the second part; and
perform weighted summation on the subword vectors corresponding to the subwords in the second part by using the mutual attention weights so as to obtain the mutual attention information of the subword in the second part.

11. The apparatus for machine reading comprehension as claimed in claim 8 ,
wherein the one or more processors are configured to
for each subword in the paragraph-question pair, fuse the self-attention information and the mutual attention information of the subword so as to obtain the fusion vector of the subword.

12. The apparatus for machine reading comprehension as claimed in claim 8 ,
wherein the one or more processors are configured to
for each subword in the paragraph-question pair, fuse the self-attention information and the mutual attention information of the subword so as to obtain attention information of the subword; and
fuse the attention information of the subword

Assignees

• Xiao Tianxiong
• Cheng Rui
• Dong Bin
• Jiang Shanshan
• Zhang Jiashi
• Ricoh Co Ltd

Inventors

• Xiao Tianxiong
• Cheng Rui
• Dong Bin
• Jiang Shanshan
• Zhang Jiashi

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06F16/3334

  Selection or weighting of terms from queries, including natural language queries · CPC title

• G06F40/56

  Natural language generation · CPC title

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title

• G06F16/3329

  Natural language query formulation · CPC title