Ingestion system for distributed graph database

Patent metadata

Field	Value
Publication number	US-12045283-B2
Application number	US-202117357925-A
Country	US
Kind code	B2
Filing date	Jun 24, 2021
Priority date	Jun 24, 2021
Publication date	Jul 23, 2024
Grant date	Jul 23, 2024

Abstract

The disclosed technologies are capable of reading ingestion configuration data for a client of a plurality of clients of a graph database, transforming the ingestion configuration data from a declarative representation into a graph representation of the ingestion configuration data, storing the graph representation of the ingestion configuration data in the graph database, providing the graph representation of the ingestion configuration data to a data service, where the data service comprises one or more of (i) a physical grouping of at least one computing device configured to store at least a portion of the graph database, (ii) a logical grouping of at least one computing device configured to store at least a portion of the graph database, or (iii) a combination of (i) and (ii).

First claim

What is claimed is:

1. A method comprising:
reading ingestion configuration data for a client of a plurality of clients of a graph database, wherein the client corresponds to a logical grouping of data stored in at least one source data store;
transforming the ingestion configuration data for the client from a declarative representation into a graph representation of the ingestion configuration data for the client;
encoding the graph representation of the ingestion configuration data for the client into the graph database to enable the creation of a self-describing image of the graph database; and
using the self-describing image of the graph database to evaluate an ingestion configuration for at least one data service of the distributed data service by:
comparing ingestion configuration data enabled for the at least one data service to ingestion configuration data applicable to the at least one data service;
detecting a mismatch between the ingestion configuration data enabled for the at least one data service and the ingestion configuration data applicable to the at least one data service; and
modifying an operation of the at least one data service based on the detected mismatch, wherein the at least one data service comprises one or more of (i) a physical grouping of at least one computing device configured to store at least a portion of the graph database, (ii) a logical grouping of at least one computing device configured to store at least a portion of the graph database, or (iii) a combination of (i) and (ii).

2. The method of claim 1 , wherein reading ingestion configuration data comprises reading a declarative representation of one or more of (i) ingestion configuration data for an offline data source for the client or (ii) ingestion configuration data for a nearline data source for the client, or (iii) a combination of (i) and (ii).

3. The method of claim 1 , wherein reading ingestion configuration data comprises reading a declarative representation of one or more of (i) location data that identifies a physical or logical location of a data source for the client, (ii) metadata that specifies an order of priority for writing data to the graph database for the client, or (iii) transformation logic for transforming input data stored in the data source into a graph representation of the input data, or (iv) a combination of at least two of (i), (ii), (iii).

4. The method of claim 1 , wherein reading ingestion configuration data comprises reading a declarative representation of one or more of (i) applicable data service data that identifies at least one data service to which the ingestion configuration data is applicable, (ii) enabled data service data that identifies at least one data service for which the ingestion configuration data is enabled, or (iii) a combination of (i) and (ii).

5. The method of claim 1 , wherein creating the graph representation of the ingestion configuration data for the client comprises transforming the ingestion configuration data into a graph primitive that comprises at least two nodes, at least one edge, and at least one predicate; wherein an edge of the at least one edge indicates a logical relationship between two nodes of the at least two nodes.

6. The method of claim 5 , wherein a predicate of the at least one predicate comprises a label for the logical relationship between the two nodes, and the label relates to an ingestion configuration, and the ingestion configuration comprises a specification for ingesting input data from a non-graph data source into the graph database.

7. The method of claim 1 , further comprising reading ingestion configuration data for the client using an application programming interface (API) configured at least in part according to a declarative query language.

8. The method of claim 1 , further comprising (i) modifying the graph representation of the ingestion configuration data to indicate that the ingestion configuration data is enabled for the at least one data service or (ii) modifying the graph representation of the ingestion configuration data to indicate that the ingestion configuration data is not enabled for the at least one data service.

9. The method of claim 1 , further comprising:
querying the self-describing image of the graph database to determine, for a client of a plurality of clients of the graph database, the ingestion configuration data applicable to a data service of a plurality of data services serving the graph database, wherein the client corresponds to least one dataset arranged according to a non-graph schema; and
reading state data for the data service to determine, for the client, the ingestion configuration data enabled for the data service.

10. The method of claim 9 , wherein comparing the ingestion configuration data enabled for the data service to the ingestion configuration data applicable to the data service comprises one or more of (i) evaluating ingestion configuration data for an offline data source for the client, (ii) evaluating ingestion configuration data for a nearline data source for the client, or (iii) a combination of (i) and (ii).

11. The method of claim 9 , wherein comparing the ingestion configuration data enabled for the data service to the ingestion configuration data applicable to the data service comprises one or more of (i) evaluating location data that identifies a physical or logical location of a non-graph data source for the client, (ii) evaluating metadata that specifies an order of priority for writing data to the graph database for the client, or (iii) a combination of (i) and (ii).

12. The method of claim 9 , wherein detecting the mismatch between the ingestion configuration data enabled for the data service and the ingestion configuration data applicable to the data service comprises determining that one or more of (i) an offline ingestion is applicable to the data service but is not enabled for the data service, (ii) a nearline ingestion is applicable to the data service but is not enabled for the data service, (iii) metadata contained in the ingestion configuration data enabled for the data service does not match metadata contained in the ingestion configuration data applicable to the data service, or (iv) at least two of (i), (ii), (iii).

13. The method of claim 9 , wherein modifying an operation of the data service based on the detected mismatch comprises one or more of (i) building an image on the data service, (ii) writing an image to a network-accessible storage location, (iii) deleting an image from the data service, (iv) copying an image to the data service from another storage location, or (v) at least two of (i), (ii), (iii), (iv).

14. The method of claim 1 , further comprising:
configuring at least one computer memory according to a logical graph of at least a portion of the graph database, wherein (i) the logical graph comprises a graph representation of ingestion configuration data, and (ii) the graph representation of the ingestion configuration data includes, for a client of a plurality of clients of the graph database, wherein the client corresponds to at least one dataset arranged according to a non-graph schema, a specification for ingesting data from at least one source data store into the graph database.

15. The method of claim 14 , wherein the graph representation of the ingestion configuration data comprises at least two nodes, at least one edge, and at least one predicate, and the at least two nodes includes one or more of (i) an offline configuration node, (ii) a nearline configuration node, or (iii) a combination of (i) and (ii).

16. The method of cl

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Sen Indranil
• Shah Siddharth
• Rodriguez Andrew M
• Sivakumar Shrinivas
• Goyal Tushar
• Sathyanarayanan Kartik
• Shah Suruchi
• Li Kenneth Y
• Ki Soyeon
• Codispoti Andrew

Classifications

• G06F9/54

  Interprogram communication · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F16/212

  with details for data modelling support · CPC title

• G06F16/9024 Primary

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title