Knowledge-graph-embedding-based question answering

Patent metadata

Field	Value
Publication number	US-11727243-B2
Application number	US-201916262618-A
Country	US
Kind code	B2
Filing date	Jan 30, 2019
Priority date	Jan 30, 2019
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

Described herein are embodiments for question answering over knowledge graph using a Knowledge Embedding based Question Answering (KEQA) framework. Instead of inferring an input questions' head entity and predicate directly, KEQA embodiments target jointly recovering the question's head entity, predicate, and tail entity representations in the KG embedding spaces. In embodiments, a joint distance metric incorporating various loss terms is used to measure distances of a predicated fact to all candidate facts. In embodiments, the fact with the minimum distance is returned as the answer. Embodiments of a joint training strategy are also disclosed for better performance. Performance evaluation on various datasets demonstrates the effectiveness of the disclosed systems and methods using the KEQA framework.

First claim

What is claimed is:

1. A computer-implemented method for question answering using one or more processors to cause steps to be performed comprising:
generating, using a predicate learning model and information of a knowledge graph (KG), a predicted predicate representation in a KG predicate embedding space for a question comprising one or more tokens;
generating, using a head entity learning model and information of the KG, a predicted head entity representation in a KG entity embedding space for the question;
obtaining a predicted tail entity representation, based on a relation function that relates, for a fact in KG embedding space, a head entity representation and a predicate representation to a tail entity representation, from the predicted predicate representation and the predicted head entity representation, wherein the predicted predicate representation, the predicted head entity representation, and the predicted tail entity representation forming a predicted fact;
identifying, using a head entity detection (HED) model, one or more predicted head entity names for the question, each predicted head entity name comprises one or more tokens from the question;
searching, in the KG, head entity synonyms related to the one or more predicted head entity names;
constructing a candidate fact set comprising one or more candidate facts, each candidate fact comprises a head entity among the head entity synonyms, a predicate, and a tail entity; and
choosing, based on a joint distance metric, one candidate fact in the candidate fact set with a minimum joint distance to the predicted fact as an answer to the question, the joint distance metric comprises:
a first distance term related to a distance between the predicted head entity representation and a head entity embedding representation for the head entity in each candidate fact;
a second distance term related to a distance between the predicted predicate representation and a predicate embedding representation for the predicate in each candidate fact; and
a third distance term related to a distance between the predicted tail entity representation and a tail entity embedding representation for the tail entity in each candidate fact.

2. The computer-implemented method of claim 1 wherein the predicate learning model has a neural network structure comprising a bidirectional recurrent neural network layer and an attention layer, and wherein the step of generating the predicted predicate representation comprises:
mapping the one or more tokens in the question into a sequence of word embedding vectors;
generating, using the bidirectional recurrent neural network layer, a forward hidden state sequence and a backward hidden state sequence;
concatenating the forward and backward hidden state vectors into a concatenated hidden state vector;
applying, by the attention layer, an attention weight to the concatenated hidden state vector to obtain a weighted hidden state vector;
concatenating the weighted hidden state vector with the word embedding to obtain a hidden state for each token;
applying a fully connected layer to the hidden state to obtain a target vector for each token; and
using a mean of all target vectors as the predicted predicate representation.

3. The computer-implemented method of claim 2 wherein the head entity learning model has a neural network structure the same as the predicate learning model.

4. The computer-implemented method of claim 3 wherein the predicate learning model and the head entity learning model are pre-trained using a training data set with ground truth facts via a predicate objective function and a head entity objective function, respectively.

5. The computer-implemented method of claim 1 wherein the HED model has a neural network structure comprising a bidirectional recurrent neural network layer and a fully connecter layer, and wherein the step of identifying the one or more predicted head entity names for the question comprises:
mapping the one or more tokens in the question into a sequence of word embedding vectors;
generating, at the bidirectional recurrent neural network layer, a forward hidden state sequence and a backward hidden state sequence;
concatenating the forward and backward hidden state vectors to obtain a concatenated hidden state vector;
applying the fully connected layer and a Softmax function to the concatenated hidden state vector to obtain a target vector for each token, each target vector has two probability values corresponding to probabilities that the token belongs to entity token name and non-entity token name; and
selecting one or more tokens as the head entity name based on probability value of each token belonging to entity token name.

6. The computer-implemented method of claim 1 wherein each of the first, second, and third distance terms uses a 2 norm to measure distance.

7. The computer-implemented method of claim 6 wherein the joint distance metric further comprises string similarity terms representing a string similarity between name of entity in each candidate fact and the tokens classified as entity name by the HED model, and a string similarity between a name of the predicate in each candidate fact and the tokens classified as non-entity name by the HED model.

8. The computer-implemented method of claim 7 wherein the joint distance metric is a weighted combination of the distance terms and the string similarity terms.

9. The computer-implemented method of claim 6 wherein in each candidate fact, the tail entity embedding representation is calculated, using a relation function, from the head entity embedding representation and the predicate embedding representation.

10. The computer-implemented method of claim 1 wherein searching head entity synonyms in the KG related to the one or more predicted head entity names comprises:
inputting each predicated head entity name into the KG; and
searching, in the KG, head entity synonyms for each predicated head entity name, by both embedding comparison and string match, wherein each head entity synonym has a direct or partial string match to the predicated head entity name or has embedding similarity to the predicated head entity name.

11. The computer-implemented method of claim 10 wherein for a predicated head entity name comprising multiple tokens, an entity representation for the predicated head entity name is formed from a dot product of entity representations of each token.

12. A computer-implemented method for question answering using one or more processors that cause steps to be performed comprising:
generating, using a predicate learning model stored in one or more memories of one or more computing devices and information of a knowledge graph (KG), a predicted predicate representation for a question comprising one or more tokens in a predicate embedding space, the predicate learning model being pre-trained using training data with ground truth facts and a predicate objective function;
generating, using a head entity learning model stored in one or more memories of one or more computing devices and information of the KG, a predicted head entity representation for the question in an entity embedding space, head entity learning model being pre-trained using training data with ground truth facts and a head entity objective function;
identifying, using a relation function based upon KG embedding, a predicted tail entity representation from the predicted predicate representation and the predicted head entity representation, wherein the predicted head entity representation, the predicted predicate representation, and the predicted tail entity representation forming a predicted fact; and
selecti

Assignees

• Baidu Usa Llc

Inventors

• Zhang Jingyuan
• Li Dingcheng
• Li Ping
• Huang Xiao

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/006 Primary

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06F16/24522

  Translation of natural language queries to structured queries · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title