System and methods for object version tracking and read-time/write-time data federation

Patent metadata

Field	Value
Publication number	US-12566740-B2
Application number	US-202418589255-A
Country	US
Kind code	B2
Filing date	Feb 27, 2024
Priority date	Dec 8, 2017
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

Systems and methods are provided in order to avoid ingesting the entire contents of one or more data sources. An object may be associated with one or more pieces/fragments of data that can be stored in different data sources. When a request for the object is initiated, a search for those pieces/fragments of data can be performed. Nodes associated with each data source may generate data queries appropriate for its corresponding data source, retrieve, and if needed transform the data into an object-based data structure. Any pieces/fragments of data that have been discovered and retrieved can be joined, e.g., by an application programming interface server, and forwarded to a requesting client or application. In this way, only data relevant to the object is obtained. Moreover, object versioning can be employed so that the most up-to-date data is obtained.

First claim

The invention claimed is:

1 . A system for analyzing data, comprising:
one or more processors, the processors comprising adapter nodes; and a memory storing instructions that, when executed by the one or more processors, cause the system to:
receive a request, from an application, for an object;
retrieve, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
the adapter nodes are intermediaries between the datastores and application programming interface (API) servers;

transform at least a portion of the data fragments, wherein the transforming comprises:
transforming a first portion of the data fragments based on a first version of a script, the first version of the script being generated according to a first schema of a datastore and a first target dataset ontology of a platform downstream of the application;
receiving an update to the first version of the script, the update resulting in generating a second version of the script, the second version of the script being generated according to the first schema or a second schema of the datastore and the first target dataset ontology or a second target dataset ontology of the platform downstream of the application; and
transforming the first portion or a second portion of the data fragments or modifying the transformed first portion of the data fragments based on the second version of the script;

upon transformation of at least the portion of the data fragments by the subset of the adapter nodes, join the data fragments by the API servers; and
transmit, by the API servers, the joined data fragments to the application.

2 . The system of claim 1 , wherein the adapter nodes further comprise respective caches in which most updated versions of the respective data fragments are stored.

3 . The system of claim 1 , further comprising a data analysis platform that stores a different data fragment; and wherein the instructions further cause the system to:
join the different data fragment to the joined data fragments to form composite data fragments, and wherein the transmitting of the joined data fragment comprises transmitting the composite data fragments.

4 . The system of claim 1 , wherein the instructions further cause the system to:
inactivate an adapter node and activate a new adapter node to replace the inactivated adapter node.

5 . The system of claim 1 , wherein, the instructions further cause the system to:
indicate that the retrieving of the data fragments has been partially completed; and transmit a partial object view to the application.

6 . The system of claim 1 , wherein the instructions further cause the system to:
receive an indication that a data fragment has updated a version; and in response to receiving the indication, reingesting, at the application, a new version of the data fragment.

7 . The system of claim 1 , wherein the joining of the data fragments comprises converting a first data fragment of a first data type by a first adapter node and a second data fragment of a second data type by a second adapter node into a standardized data type.

8 . The system of claim 1 , wherein
receive, from the datastores, a new version of a data fragment within a datastore; in response to receiving the new version of the data fragment, instantiating a new adapter node to retrieve the new version of the data fragment; and retrieve, from the datastore, using the new adapter node, the new version of the data fragment; transform at least a portion of the new version of the data fragment based on the first version or the second version of the script; and upon transformation of at least a portion of the new version of the data fragment, join the new version of the data fragment to other data fragments by the API servers; and transmit, by the API servers, the joined new version of the data fragment and the other data fragments.

9 . A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
receiving a request, from an application, for an object; retrieving, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
the adapter nodes are intermediaries between the datastores and application programming interface (API) servers;
transforming at least a portion of the data fragments, wherein the transforming comprises:
transforming a first portion of the data fragments based on a first version of a script, the first version of the script being generated according to a first schema of a datastore and a first target dataset ontology of a platform downstream of the application;
receiving an update to the first version of the script, the update resulting in generating a second version of the script, the second version of the script being generated according to the first schema or a second schema of the datastore and the first target dataset ontology or a second target dataset ontology of the platform downstream of the application; and
transforming the first portion or a second portion of the data fragments or modifying the transformed first portion of the data fragments based on the second version of the script;
upon transformation of at least the portion of the data fragments by the subset of the adapter nodes, joining the data fragments by the API servers; and transmitting, by the API servers, the joined data fragments to the application.

10 . The method of claim 9 , further comprising storing most updated versions of the respective data fragments in respective caches of the adapter nodes.

11 . The method of claim 9 , further comprising:
storing a different data fragment in a data analysis platform; and joining the different data fragment to the joined data fragments to form composite data fragments, wherein the transmitting of the joined data fragment comprises transmitting the composite data fragments.

12 . The method of claim 9 , further comprising:
inactivating an adapter node and activating a new adapter node to replace the inactivated adapter node.

13 . The method of claim 9 , further comprising:
indicating that the retrieving of the data fragments has been partially completed; and transmitting a partial object view to the application.

14 . The method of claim 9 , wherein a first of the data fragments comprises a tabular format and a second of the data fragments comprises a second, non-object oriented format.

15 . The method of claim 9 , further comprising:
receiving an indication that a data fragment has updated a version; and in response to receiving the indication, reingesting, at the application, a new version of the data fragment.

16 . A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
receiving a request, from an application, for an object; retrieving, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
the adapter nodes are intermediaries between the datastores and application programming interface (API) servers;
transforming at least a portion of the data fragments, wherein the transforming comprises:
transforming a first portion of the data fragments based on a first version of a script, the first version of the script being generated according to a first schema of a datastore and a first target dataset ontology of a platform downstream of the app

Assignees

• Palantir Technologies Inc

Inventors

• Wilczynski Peter
• Ryan Alexander
• Chang Allen
• Cervelli Daniel
• Tibshirani Julie
• Wilson Timothy

Classifications

• G06F16/2456

  Join operations · CPC title

• G06F16/256

  in federated or virtual databases · CPC title

• G06F16/2474

  Sequence data queries, e.g. querying versioned data · CPC title

• G06F16/219 Primary

  Managing data history or versioning (querying versioned data G06F16/2474; querying temporal data G06F16/2477) · CPC title

• G06F16/2458 Primary

  Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries · CPC title