System and method of efficient knowledge-enhanced chain-of-thought prompting

Patent metadata

Field	Value
Publication number	US-12596731-B2
Application number	US-202418407027-A
Country	US
Kind code	B2
Filing date	Jan 8, 2024
Priority date	Jan 8, 2024
Publication date	Apr 7, 2026
Grant date	Apr 7, 2026

Abstract

Methods and systems for processing a natural language input query that includes a question and a respective set of candidate answers for the question, including generating, based on the input query and a knowledge graph, natural language logic paths between at least some of the candidate answers and the question; forming a natural language prompt based on both the input query and the logic paths; and obtaining a response from a pretrained natural language processing model based on the natural language promp.

First claim

The invention claimed is:

1 . A computer-implemented method for processing a natural language input query that includes a question and a respective set of candidate answers for the question, the method comprising:
generating, based on the input query and a knowledge graph, natural language logic paths between at least some of the candidate answers and the question, the generating includes:
identifying a question entity for the question included in the input query;
identifying a respective candidate answer entity for each of the candidate answers included in the input query;
identifying, for the question entity, a matching question node entity of the knowledge graph;
identifying, for at least some of the candidate answer entities, respective matching node answer entities of the knowledge graph; and
identifying, based on inter-node entity relationships specified in the knowledge graph, a respective logic path for each matching node answer entity to the matching question node entity as the natural language logic paths;
forming a natural language prompt based on both the input query and the logic paths; and obtaining a response, based on the natural language prompt, from a pretrained natural language processing model based on a large neural network operating under a pretrain-finetune paradigm.

2 . The method of claim 1 wherein:
identifying the matching question node entity for the question entity is based on a comparison of an embedding generated for the question entity with embeddings generated for node entities of the knowledge graph; and
identifying the respective matching node answer entities for the at least some of the candidate answer entities is based on a comparison of embeddings generated for the candidate answer entities with embeddings generated for the node entities of the knowledge graph,
the embeddings each being generated by a neural network model that has been pretrained to generate similar embeddings for terms having semantically similar meanings.

3 . The method of claim 2 wherein identifying the respective logic path for each matching node answer entity to the matching question node entity comprises:
identifying, for each matching node answer entity, a respective set of candidate logic paths for the matching node answer entity to the matching question node entity;
ranking each of the candidate logic paths within each of the respective sets of candidate logic paths; and
selecting as the respective logic path for each matching node answer entity the candidate logic path having the highest ranking from the respective set of candidate logic paths for the matching node answer entity.

4 . The method of claim 3 wherein identifying the respective set of candidate logic paths for each matching node answer entity is limited to paths that fall within a predefined number of entity node hops.

5 . The method of claim 1 wherein identifying the question entity for the question comprises applying a keyphrase extraction tool to the question to extract the question entity.

6 . The method of claim 1 wherein forming the natural language prompt comprises combining content from the input query, the logic paths and one or more exemplars to form the natural language prompt, wherein each exemplar includes a representation of an example natural language prompt combined with an example response thereto, each example natural language prompt comprising a respective example question and set of example candidate answers together with example logic paths between the example question and the example candidate answers.

7 . The method of claim 1 wherein the response indicates a selected answer from the set of candidate answers and a natural language chain-of-thought statement for the selected answer.

8 . The method of claim 1 comprising receiving the input query over a network from a requesting device and providing the response over the network to the requesting device.

9 . The method of claim 1 comprising receiving the knowledge graph over a network from a requesting device.

10 . A computing system comprising:
a processing unit configured to execute computer-readable instructions to cause the system to perform a method for processing a natural language input query that includes a question and a respective set of candidate answers for the question, the method including: generating, based on the input query and a knowledge graph, natural language logic paths between at least some of the candidate answers and the question, the generating includes:
identifying a question entity for the question included in the input query;
identifying a respective candidate answer entity for each of the candidate answers included in the input query;
identifying, for the question entity, a matching question node entity of the knowledge graph;
identifying, for at least some of the candidate answer entities, respective matching node answer entities of the knowledge graph; and
identifying, based on inter-node entity relationships specified in the knowledge graph, a respective logic path for each matching node answer entity to the matching question node entity as the natural language logic paths;
forming a natural language prompt based on both the input query and the logic paths; and obtaining a response, based on the natural language prompt, from a pretrained natural language processing model based on a large neural network operating under a pretrain-finetune paradigm.

11 . The computing system of claim 10 wherein:
identifying the matching question node entity for the question entity is based on a comparison of an embedding generated for the question entity with embeddings generated for node entities of the knowledge graph; and
identifying the respective matching node answer entities for the at least some of the candidate answer entities is based on a comparison of embeddings generated for the candidate answer entities with embeddings generated for the node entities of the knowledge graph,
the embeddings each being generated by a neural network model that has been pretrained to generate similar embeddings for terms having semantically similar meanings.

12 . The computing system of claim 11 wherein identifying the respective logic path for each matching node answer entity to the matching question node entity comprises:
identifying, for each matching node answer entity, a respective set of candidate logic paths for the matching node answer entity to the matching question node entity;
ranking each of the candidate logic paths within each of the respective sets of candidate logic paths; and
selecting as the respective logic path for each matching node answer entity the candidate logic path having the highest ranking from the respective set of candidate logic paths for the matching node answer entity.

13 . The computing system of claim 12 wherein identifying the respective set of candidate logic paths for each matching node answer entity is limited to paths that fall within a predefined number of entity node hops.

14 . The computing system of claim 10 wherein identifying the question entity for the question comprises applying a keyphrase extraction tool to the question to extract the question entity.

15 . The computing system of claim 10 wherein forming the natural language prompt comprises combining content from the input query, the logic paths and one or more exemplars to form the natural language prompt, wherein each exemplar includes a representation of an example natural language prompt combined with an example response thereto, each example natural language prompt comprising a respec

Assignees

• Heisler Morgan Lindsay
• Wang Chendi
• Singh Gursimran
• Zhang Yong
• Huawei Cloud Computing Tech Co Ltd

Inventors

• Heisler Morgan Lindsay
• Wang Chendi
• Singh Gursimran
• Zhang Yong

Classifications

• G06F16/3341

  using boolean model · CPC title

• G10L15/30

  Distributed recognition, e.g. in client-server systems, for mobile phones or network applications · CPC title

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F16/3329 Primary

  Natural language query formulation · CPC title