Using an ontologically typed graph to enhance the accuracy of a large language model based analysis system

Patent metadata

Field	Value
Publication number	US-2024411797-A1
Application number	US-202318208840-A
Country	US
Kind code	A1
Filing date	Jun 12, 2023
Priority date	Jun 12, 2023
Publication date	Dec 12, 2024
Grant date	—

Abstract

A large language model consumes example query expressions, including a data access function, a data analytics function, or a data enrichment function. The large language model receives a centrally managed ontology. The large language model uses the centrally managed ontology, and identifies skill ontological types from the example query expressions. The skill ontological types are normalized (to the centrally managed ontology) input arguments types or structured output. The large language model receives context for an investigation and identifies a context ontological type. The large language model receives received skills, based on correlation between a skill ontological type, having connections in a graph to the received skills, and the context ontological type. The large language model produces and provides an indication of a suggested skill for the investigation.

First claim

What is claimed is:

1 . A method comprising:
at a computing system, using network crawling to identify a plurality of example query expressions stored at various different locations on a network, the example query expressions in the plurality of example query expressions having one or more specific data access functions, specific data analytics functions, or specific data enrichment functions, and having specific arguments; storing the plurality of example query expressions in storage at the computing system; transmitting, over a network connection the example query expressions to a large language model system; providing from the computing system, over the network connection, a centrally managed ontology to the large language model system; receiving from the large language model system, a plurality of annotated skills, the annotated skills in the plurality of annotated skills being genericized versions of example query expressions in the plurality of example query expressions, the annotated skills comprising skill ontological types genericized from entities the example query expressions in the plurality of example query expressions, the skill ontological types being related to at least one of input arguments to the annotated skills or structured output of the annotated skills, the skill ontological types being normalized to the centrally managed ontology; using the skill ontological types, storing an ontologically typed graph having skills in the plurality of skills coupled to each other through in-common, normalized, ontological types; providing, over the network connection, at least a portion of the plurality of the annotated skills in the ontologically typed graph to the large language model system; and receiving over the network, from the large language model system a message indicating an investigation skill to be invoked.

2 . The method of claim 1 , further comprising:
providing context to the large language model system; receiving from the large language model system a context ontological type, the context ontological type being normalized to the centrally managed ontology; using the context ontological type, pruning the ontologically typed graph to store a pruned graph.

3 . The method of claim 2 , wherein providing context to the large language model system comprises providing an initial context and investigation goal to the large language model system.

4 . The method of claim 2 , wherein providing at least a portion of the plurality of annotated skills in the ontologically typed graph is performed by providing the pruned graph, and wherein the investigation skill to be invoked is from the pruned graph.

5 . The method of claim 2 , further comprising:
invoking a skill; and wherein providing context to the large language model system comprises providing a context created as a result of invoking the skill.

6 . The method of claim 1 , wherein the plurality of example query expressions comprises functionality for generating a log.

7 . A method comprising:
at a large language model system, consuming a plurality of example query expressions, the example query expressions in the plurality of example query expressions comprising one or more specific data access functions, specific data analytics functions, or specific data enrichment functions, and having specific arguments; at the large language model system receiving a centrally managed ontology; the large language model system identifying skill ontological types from the example query expressions, the skill ontological types being related to at least one of input arguments or structured output, the skill ontological types being normalized to the centrally managed ontology, and genericized; the large language model system generating a plurality of annotated skills, the annotated skills in the plurality of annotated skills being genericized versions the example query expressions, the annotated skills comprising skill ontological types genericized from query expression; the large language model system providing the plurality of annotated skills, over a network connection, to an external computing system; the large language model system receiving context for an investigation; the large language model system identifying a context ontological type from the context, using the centrally managed ontology; the large language model providing the context ontological type to the computing system, over the network connection; the large language model system receiving received skills, from the plurality of annotated skills, over the network, from the computing system, based on correlation between a skill ontological type and the context ontological type; and as a result, the large language model system, using the trained model, producing and providing a message of a suggested skill for the investigation.

8 . The method of claim 7 , wherein receiving context for the investigation; identifying the context ontological type from the context; receiving received skills; and providing an indication of suggested skills for the investigation are performed recursively.

9 . The method of claim 7 , further comprising receiving a schema, and wherein generating a plurality of annotated skills is performed using the schema.

10 . The method of claim 7 , the large language model system receiving an investigation goal, and wherein providing the indication of the suggested skill for the investigation is performed using the investigation goal.

11 . The method of claim 7 , wherein at least one of the plurality of example query expressions is configured to generate a log.

12 . The method of claim 7 , wherein at least one of the plurality of example query expressions is configured to generate a table.

13 . The method of claim 7 , wherein at least one of the plurality of example query expressions is configured to generate a database view.

14 . The method of claim 7 , wherein at least one of the plurality of example query expressions is configured to invoke an API skill.

15 . The method of claim 7 , wherein providing an indication of the suggested skill for the investigation comprises providing an indication of a combined query comprising a plurality of skills invoked together.

16 . The method of claim 7 , wherein identifying the skill ontological types or the context ontological type comprises adapting native ontological types to normalize the skill ontological types to the centrally managed ontology.

17 . The method of claim 7 , further comprising performing a shortest path analysis, and wherein providing an indication of the suggested skill for the investigation comprises providing an indication of a skills identified in the shortest path analysis.

18 . A computing system comprising:
a processor; and computer-readable media having stored thereon instructions that are executable by the processor to configure the computer system to normalize ontological types in skills and to automatically receive skill recommendation messages, including instructions that are executable to configure the computer system to perform at least the following: at the computing system, network crawling to identify a plurality of example query expressions stored at various different locations on a network, the example query expressions in the plurality of example query expressions having one or more specific data access functions, specific data analytics functions, or specific data enrichment functions, and having specific arguments; store the plurality of example query expressions in storage at the computi

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Blum William
• Diotte Sébastien Martin
• Fontaine Martin Jean
• Sobey Edward Richard
• Maor Tal Joseph
• Ranjit Shruti
• Donaghy Richard
• Ezrets Gil Michal
• Clowes Cory James
• Makedon Emily Maria
• Wilyman Ross Kingsley

Classifications

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title

• G06F16/355

  Creation or modification of classes or clusters · CPC title

• G06F16/338

  Presentation of query results · CPC title

• G06F16/332

  Query formulation · CPC title

• G06F16/367 Primary

  Ontology · CPC title