System and methods for object version tracking and read-time/write-time data federation

Patent metadata

Field	Value
Publication number	US-11914558-B2
Application number	US-202217825549-A
Country	US
Kind code	B2
Filing date	May 26, 2022
Priority date	Dec 8, 2017
Publication date	Feb 27, 2024
Grant date	Feb 27, 2024

Abstract

Systems and methods are provided in order to avoid ingesting the entire contents of one or more data sources. An object may be associated with one or more pieces/fragments of data that can be stored in different data sources. When a request for the object is initiated, a search for those pieces/fragments of data can be performed. Nodes associated with each data source may generate data queries appropriate for its corresponding data source, retrieve, and if needed transform the data into an object-based data structure. Any pieces/fragments of data that have been discovered and retrieved can be joined, e.g., by an application programming interface server, and forwarded to a requesting client or application. In this way, only data relevant to the object is obtained. Moreover, object versioning can be employed so that the most up-to-date data is obtained.

First claim

The invention claimed is:

1. A system for analyzing data, comprising:
one or more processors, the processors comprising adapter nodes; and
a memory storing instructions that, when executed by the one or more processors, cause the system to:
receive a request, from an application, for an object;
retrieve, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
each datastore corresponds to a particular data type,
at least one datastore has a different data type compared to a data type of another datastore;
the adapter nodes are intermediaries between the datastores and application programming interface (API) servers, the adapter nodes being specifically generated to retrieve data of a particular data type; and
the retrieving comprises:
determining, by each of the subset of the adapter nodes, a query format to search a datastore according to the particular data type of the datastore;
generating, by each of the subset of the adapter nodes, a query to search the datastore based on the determined query format;
retrieving, by each of the subset of the adapter nodes based on the generated query, a data fragment in one of the datastores; and
transforming at least a portion of the data fragments, wherein the transforming is based on a script, the script being generated according to a schema of a datastore from which a particular data fragment of the portion of the data fragments is retrieved and a target ontology of the application or a platform;

upon transformation of at least the portion of the data fragments by the subset of the adapter nodes, join the data fragments by the API servers; and
transmit, by the API servers, the joined data fragments to the application.

2. The system of claim 1 , wherein the adapter nodes further comprise respective caches in which most updated versions of the respective data fragments are stored.

3. The system of claim 1 , further comprising a data analysis platform that stores a different data fragment; and wherein the instructions further cause the system to:
join the different data fragment to the joined data fragments to form composite data fragments, and wherein the transmitting of the joined data fragment comprises transmitting the composite data fragments.

4. The system of claim 1 , wherein the instructions further cause the system to:
inactivate an adapter node and activate a new adapter node to replace the inactivated adapter node.

5. The system of claim 1 , wherein, the instructions further cause the system to:
indicate that the retrieving of the data fragments has been partially completed; and
transmit a partial object view to the application.

6. The system of claim 1 , wherein the instructions further cause the system to:
receive an indication that a data fragment has updated a version; and
in response to receiving the indication, reingesting, at the application, a new version of the data fragment.

7. The system of claim 1 , wherein the instructions further cause the system to:
in response to receiving the request for the object, generate the adapter nodes.

8. The system of claim 1 , wherein the joining of the data fragments comprises converting a first data fragment of a first data type by a first adapter node and a second data fragment of a second data type by a second adapter node into a standardized data type.

9. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
receiving a request, from an application, for an object;
retrieving, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
each datastore corresponds to a particular data type,
at least one datastore has a different data type compared to a data type of another datastore;
the adapter nodes are intermediaries between the datastores and application programming interface (API) servers, the adapter nodes being specifically generated to retrieve data of a particular data type; and
the retrieving comprises:
determining, by each of the subset of the adapter nodes, a query format to search a datastore according to the particular data type of the datastore;
generating, by each of the subset of the adapter nodes, a query to search the datastore based on the determined query format;
retrieving, by each of the subset of the adapter nodes based on the generated query, a data fragment in one of the datastores; and
transforming at least a portion of the data fragments, wherein the transforming is based on a script, the script being generated according to a schema of a datastore from which a particular data fragment of the portion of the data fragments is retrieved and a target ontology of the application or a platform;
upon transformation of at least the portion of the data fragments by the subset of the adapter nodes, joining the data fragments by the API servers; and

transmitting, by the API servers, the joined data fragments to the application.

10. The method of claim 9 , further comprising storing most updated versions of the respective data fragments in respective caches of the adapter nodes.

11. The method of claim 9 , further comprising:
storing a different data fragment in a data analysis platform; and
joining the different data fragment to the joined data fragments to form composite data fragments, wherein the transmitting of the joined data fragment comprises transmitting the composite data fragments.

12. The method of claim 9 , further comprising:
inactivating an adapter node and activating a new adapter node to replace the inactivated adapter node.

13. The method of claim 9 , further comprising:
indicating that the retrieving of the data fragments has been partially completed; and
transmitting a partial object view to the application.

14. The method of claim 9 , wherein a first of the data fragments comprises a tabular format and a second of the data fragments comprises a second, non-object oriented format.

15. The method of claim 9 , further comprising:
receiving an indication that a data fragment has updated a version; and
in response to receiving the indication, reingesting, at the application, a new version of the data fragment.

16. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
receiving a request, from an application, for an object;
retrieving, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
each datastore corresponds to a particular data type,
at least one datastore has a different data type compared to a data type of another datastore;
the adapter nodes are separate from the datastores and are intermediaries between the datastores and application programming interface (API) servers, the adapter nodes being specifically generated to retrieve data of a particular data type; and
the retrieving comprises:
determining, by each of the subset of the adapter nodes, a query format to search a datastore according to the particular data type of the datastore;
generating, by each of the subset of the adapter nodes, a query to search the datastore based on the determined query format;
retrieving, by each of the subset of the adapter nodes based on the generated query, a data fragment in one of the datastores; and
transforming at least a portion of the data fragments, wherein the transforming is based on a scr

Assignees

• Palantir Technologies Inc

Inventors

• Wilczynski Peter
• Ryan Alexander
• Chang Allen
• Cervelli Daniel
• Tibshirani Julie
• Wilson Timothy

Classifications

• G06F16/219 Primary

  Managing data history or versioning (querying versioned data G06F16/2474; querying temporal data G06F16/2477) · CPC title

• G06F16/2329

  using versioning · CPC title

• G06F16/2456

  Join operations · CPC title

• G06F16/2474

  Sequence data queries, e.g. querying versioned data · CPC title

• G06F16/2458 Primary

  Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries · CPC title