Method and system for relation learning by multi-hop attention graph neural network

Patent metadata

Field	Value
Publication number	US-12299566-B2
Application number	US-202117327867-A
Country	US
Kind code	B2
Filing date	May 24, 2021
Priority date	Sep 23, 2020
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

System and method for completing knowledge graph. The system includes a computing device, the computing device has a processer and a storage device storing computer executable code. The computer executable code is configured to: provide an incomplete knowledge graph comprising a plurality of nodes and a plurality of edges, each of the edges connecting two of the plurality of nodes; calculate an attention matrix of the incomplete knowledge graph based on one-hop attention between any two of the plurality of the nodes that are connected by one of the plurality of the edges; calculate multi-head diffusion attention for any two of the plurality of nodes from the attention matrix; obtain updated embedding of the incomplete knowledge graph using the multi-head diffusion attention; and update the incomplete knowledge graph to obtain updated knowledge graph based on the updated embedding.

First claim

What is claimed is:

1. A system comprising a computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein the computer executable code, when executed at the processor, is configured to:
provide an incomplete knowledge graph comprising a plurality of nodes and a plurality of edges, each of the edges connecting two of the plurality of nodes;
calculate an attention matrix of the incomplete knowledge graph based on one-hop attention between any two of the plurality of the nodes that are connected by one of the plurality of the edges;
calculate multi-head diffusion attention for any two of the plurality of nodes from the attention matrix;
obtain an updated embedding of the incomplete knowledge graph using the multi-head diffusion attention, and train a neural network model based on the updated embedding and labels of the incomplete knowledge graph, to obtain a trained neural network model; and
update the incomplete knowledge graph to obtain updated knowledge graph by using the trained neural network model,
wherein the computer executable code is configured to calculate the attention matrix by: calculating an attention score s i,k,j (l) for an edge (v i , r k , v j ) by s i,k,j (l) =LeakyReLU (v α (l) tan h(W h (l) h i (l) ∥W t (l) h j (l) ∥W r (l) r k )) (equation (1), wherein v i and v j are nodes i and j, r k is a type of the edge between the nodes i and j, W h (l) ∈ d (l) ×d (l) , W t (l) ∈ d (l) ×d (l) , W r (l) ∈ d (l) ×d r and v α (l) ∈ 1×3d (l) are trainable weights shared by an l-th layer of the multi-head attention, h i (l) ∈ d (l) represents embedding of the node i at the l-th layer, h j (l) ∈ d (l) represents embedding of the node j at the l-th layer, r k is trainable relation embedding of a k-th relation type, and ∥ denotes concatenation of embedding vectors;
obtaining attention score matrix S (l) by:

S

i
,
j

(
l
)

=

{

s

i
,
k
,
j

(
l
)

,

if
⁢

⁢

(

v
i

,

r
k

,

v
j

)

⁢

⁢
appears
⁢

⁢
in
⁢

⁢
𝒢

-
∞

,
otherwise

,

(

equation
⁢

⁢

(
2
)

)

wherein G is the knowledge graph; and
calculating the attention matrix A (l) by: A (l) =softmax(S (l) ).

2. The system of claim 1 , wherein the computer executable code is configured to calculate the multi-head diffusion attention by:
calculating multi-hop attention matrix A by: A=Σ hop=0 ∞ θ hop A hop (equation (3)), wherein hop is a positive integer in a range of 2-20, and θ hop is an attention decay factor; and
calculating the multi-head diffusion attention by: AttDiffusion (G, H (l) , Θ)=AH (l) (equation (4)), wherein Θ represents parameters for equation (1), and H (l) is input entity embedding of the l-th layer.

3. The system of claim 2 , wherein the AH (l) is approximated by:
letting Z (0) =H (l) , Z (k+1) =(1−α)AZ (k) +αZ (0) (equation (5)), wherein 0≤k≤K, and θ hop =α(1−α) hop ; and
defining the AH (l) as Z (K) .

4. The system of claim 3 , wherein the hop or K is a positive integer in a range of 2-12, and l is a positive integer in a range of 2-24.

5. The system of claim 4 , wherein the computer executable code is configured to obtain the updated embedding of the incomplete knowledge graph by: performing sequentially a first layer normalization and addition, a feed forward, and a second layer normalization and addition on the multi-head diffusion attention.

6. The system of claim 5 , wherein the feed forward is performed using a two-layer multilayer perceptron (MLP) network.

7. The system of claim 1 , wherein the computer executable code is further configured to, after obtaining the updated embedding: calculate a loss function based on the updated embeddin

Assignees

• Beijing Wodong Tianjun Information Technology Co Ltd
• Jd Com American Tech Corp
• Univ Leland Stanford Junior

Inventors

• Wang Guangtao
• Ying Zhitao
• Huang Jing
• Leskovec Jurij

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06N3/10

  Interfaces, programming languages or software development kits, e.g. for simulating neural networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title