Efficient and compact text matching system for sentence pairs

Patent metadata

Field	Value
Publication number	US-11995111-B2
Application number	US-202017097589-A
Country	US
Kind code	B2
Filing date	Nov 13, 2020
Priority date	Nov 13, 2020
Publication date	May 28, 2024
Grant date	May 28, 2024

Abstract

A method, computer program, and computer system is provided for query matching of sentences based on co-attention scores. Two or more query inputs to a neural network are received. A correlation factor between the two or more query input is identified based on attention weights. A feature vector is generated based on the attention weights. A probability value corresponding to the two or more query inputs is determined based on the generated feature vector.

First claim

What is claimed is:

1. A method of query matching, executable by a processor, comprising:
receiving two or more query inputs to a neural network;
identifying a correlation factor between the two or more query inputs based on a multi-layer co-attention module including attention weights that are generated based on a correlation between pairs of the two or more query inputs, wherein the multi-layer co-attention module includes a plurality of layers in which each layer includes a different weighting strategy, wherein the correlation is calculated based on co-attention weight matrices capturing a concentration or focus between the pairs of the two or more query inputs, wherein each layer of the multi-layer co-attention module includes different attention weights for the co-attention weight matrices such that the co-attention weight matrices correspond to a different concentration or focus at each layer of the multi-layer co-attention module, wherein a relevance matrix is generated for each layer of the multi-layer co-attention module based on the co-attention weight matrices;
generating a feature vector based on the attention weights and the relevance matrix of each layer of the multi-layer co-attention module; and
determining, based on the generated feature vector, a probability value corresponding to a likelihood the two or more query inputs are matched.

2. The method of claim 1 , further comprising determining a pairing between two of the two or more query inputs based on the determined probability value.

3. The method of claim 1 , wherein a pair of query inputs is selected based on a loss value corresponding to the probability value being minimized.

4. The method of claim 1 , wherein the attention weights are calculated as average values based on the different weighting strategies included in the plurality of layers of the multi-layer co-attention module.

5. A computer system for query matching, the computer system comprising:
one or more computer-readable non-transitory storage media configured to store computer program code; and
one or more computer processors configured to access said computer program code and operate as instructed by said computer program code, said computer program code including:
receiving code configured to cause the one or more computer processors to receive two or more query inputs to a neural network;
identifying code configured to cause the one or more computer processors to identify a correlation factor between the two or more query inputs based on a multi-layer co-attention module including attention weights, that are generated based on a correlation between pairs of the two or more query inputs, wherein the multi-layer co-attention module includes a plurality of layers in which each layer includes a different weighting strategy, wherein the correlation is calculated based on co-attention weight matrices capturing a concentration or focus between the pairs of the two or more query inputs, wherein each layer of the multi-layer co-attention module includes different attention weights for the co-attention weight matrices such that the co-attention weight matrices correspond to a different concentration or focus at each layer of the multi-layer co-attention module, wherein a relevance matrix is generated for each layer of the multi-layer co-attention module based on the co-attention weight matrices;
generating code configured to cause the one or more computer processors to generate a feature vector based on the attention weights and the relevance matrix of each layer of the multi-layer co-attention module; and
first determining code configured to cause the one or more computer processors to determine, based on the generated feature vector, a probability value corresponding to a likelihood the two or more query inputs are matched.

6. The computer system of claim 5 , further comprising second determining code configured to cause the one or more computer processors to determine a pairing between two of the two or more query inputs based on the determined probability value.

7. The computer system of claim 5 , wherein a pair of query inputs is selected based on a loss value corresponding to the probability value being minimized.

8. The computer system of claim 5 , wherein the attention weights are calculated as average values based on the different weighting strategies included in the plurality of layers of the multi-layer co-attention module.

9. A non-transitory computer readable medium having stored thereon a computer program for query matching, the computer program configured to cause one or more computer processors to:
receive two or more query inputs to a neural network;
identifying a correlation factor between the two or more query inputs based on a multi-layer co-attention module including attention weights that are generated based on a correlation between pairs of the two or more query inputs, wherein the multi-layer co-attention module includes a plurality of layers in which each layer includes a different weighting strategy, wherein the correlation is calculated based on co-attention weight matrices capturing a concentration or focus between the pairs of the two or more query inputs, wherein each layer of the multi-layer co-attention module includes different attention weights for the co-attention weight matrices such that the co-attention weight matrices correspond to a different concentration or focus at each layer of the multi-layer co-attention module, wherein a relevance matrix is generated for each layer of the multi-layer co-attention module based on the co-attention weight matrices;
generating a feature vector based on the attention weights and the relevance matrix of each layer of the multi-layer co-attention module; and
determining, based on the generated feature vector, a probability value corresponding to a likelihood the two or more query inputs are matched.

10. The computer readable medium of claim 9 , further comprising determining code configured to cause the one or more computer processors to determine a pairing between two of the two or more query inputs based on the determined probability value.

11. The computer readable medium of claim 9 , wherein a pair of query inputs is selected based on a loss value corresponding to the probability value being minimized.

Assignees

• Tencent America LLC

Inventors

• You Zeyu
• Yang Tao
• Zhang Shangqing
• Tu min
• Wang Xu
• Han Lianyi
• Fan Wei

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06F16/3329 Primary

  Natural language query formulation · CPC title

• G06F16/3334

  Selection or weighting of terms from queries, including natural language queries · CPC title

• G06F16/3346

  using probabilistic model · CPC title