System and methods for object version tracking and read-time/write-time data federation

Patent metadata

Field	Value
Publication number	US-11347703-B1
Application number	US-201816203167-A
Country	US
Kind code	B1
Filing date	Nov 28, 2018
Priority date	Dec 8, 2017
Publication date	May 31, 2022
Grant date	May 31, 2022

Abstract

Systems and methods are provided in order to avoid ingesting the entire contents of one or more data sources. An object may be associated with one or more pieces/fragments of data that can be stored in different data sources. When a request for the object is initiated, a search for those pieces/fragments of data can be performed. Nodes associated with each data source may generate data queries appropriate for its corresponding data source, retrieve, and if needed transform the data into an object-based data structure. Any pieces/fragments of data that have been discovered and retrieved can be joined, e.g., by an application programming interface server, and forwarded to a requesting client or application. In this way, only data relevant to the object is obtained. Moreover, object versioning can be employed so that the most up-to-date data is obtained.

First claim

The invention claimed is:

1. A system for analyzing data, comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to:
receive a request for an object in accordance with a data analysis procedure;
retrieve data fragments making up or associated with the requested object, wherein each of the data fragments are stored in different data stores, and wherein each of the different data stores are operatively connected to adapter nodes configured to retrieve the data fragments from the respective data stores by identifying the data fragments from an object ID included in the request and determining an appropriate query to search for the data fragments from each of the data stores;
determine a frequency at which a data store storing the object is queried;
based on the frequency, determine whether to adjust a response time of retrieving the data fragments associated with the object;
determine whether to preload the data fragments associated with the object based on the determination of whether to adjust the response time;
selectively preload at least a portion of the data associated with the data store based on the determination of whether to preload the data fragments;
track, by the adapter nodes, metadata and versions of each of the data fragments over time to detect whether a data fragment of the data fragments has become updated;
in response to detecting that a data fragment has become updated, retrieve the data fragment that has become updated and at least an other data fragment of the data fragments;
upon retrieval of the updated data fragment and the other data fragment, join the updated data fragment and the other data fragment to create a canonical view of the object;
and transmit the canonical view of the object to an application from which the request was received.

2. The system of claim 1 , further comprising a data analysis platform through which a user of the application performs at least one of an editing procedure, a resolution procedure, and a tagging procedure related to the object.

3. The system of claim 2 , wherein the data analysis platform comprises a database storing a third data fragment of the two or more data fragments.

4. The system of claim 1 , wherein the data analysis platform comprises a persistent cache implementing a smart versioning function to maintain a most up-to-date version of the third data fragment.

5. The system of claim 4 , wherein the instructions further cause the system to query the persistent cache of the data analysis platform prior to querying the data analysis platform.

6. The system of claim 1 , wherein a first of the data fragments comprises a different type of data than a second of the data fragments.

7. The system of claim 1 , wherein at least one of the data stores maintains a corresponding one of the data fragments in a tabular format.

8. The system of claim 7 , wherein a corresponding one of the adapter nodes associated with the at least one of the data stores transforms the corresponding one of the data fragments from the tabular format to an object-based data structure.

9. The system of claim 1 , wherein each of the adapter nodes comprises or is operatively connected to a data crawler configured to determine a query format appropriate for querying a corresponding data store.

10. The system of claim 1 , wherein the instructions further cause the system to provide one or more application programming interfaces allowing the application to operatively communicate with the data stores.

11. The system of claim 1 , wherein the instructions causing the system to join the two or more data fragments further causes the system to append an ID identifying the object to each of the data fragments within the corresponding data stores.

12. The system of claim 1 , wherein the instructions cause the system to retrieve the data fragments only at a read-time or write-time operation at the application.

13. The system of claim 1 , wherein each of the adapter nodes comprises a persistent cache implementing a smart versioning function to maintain a most up-to-date version of each of the data fragments.

14. The system of claim 13 , wherein the instructions further cause the system to query the persistent caches of each of the adapter nodes prior to querying the corresponding data stores.

15. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
receiving a request for an object in accordance with a data analysis procedure;
receiving, at one of a read-time or write-time operation of the data analysis procedure, data associated with the object, the data comprising data fragments, wherein each of the data fragments are stored in different data stores, wherein the receiving comprises identifying the data fragments from an object ID included in the request and determining an appropriate query to search for the data fragments from each of the data stores;
determining a frequency at which a data store storing the object is queried;
based on the frequency, determining whether to adjust a response time of retrieving the data fragments associated with the object;
determining whether to preload the data fragments associated with the object based on the determination of whether to adjust the response time;
selectively preloading at least a portion of the data associated with the data store based on the determination of whether to preload the data fragments;
tracking, by adapter nodes, metadata and versions of each of the data fragments over time to detect whether a data fragment of the data fragments has become updated;
in response to detecting that a data fragment has been updated, retrieving the data fragment that has become updated and at least an other data fragment of the data fragments, wherein the updated data fragment and the other data fragment have been joined to create a canonical view of the object upon retrieval of the updated data fragment and the other data fragment.

16. The method of claim 15 , wherein the method further comprises formulating queries for the data fragments in accordance with the respective data stores from which the data fragments originate.

17. The method of claim 15 , wherein the method further comprises transforming each of the data fragments from a first data format into an object-based data structure commensurate with that associated with the data analysis procedure.

18. The method of claim 15 , wherein the joining of the data fragments comprises appending an ID identifying the object to each of the data fragments within the corresponding data stores.

19. The method of claim 15 , wherein the method further comprises querying smart versioning caches associated with each of the data stores prior to querying the data stores to obtain, if present in the smart versioning caches, the data fragments.

20. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
receiving a request for an object in accordance with a data analysis procedure;
retrieving data fragments making up or associated with the requested object, wherein each of the data fragments are stored in different data stores, and wherein each of the different data stores are operatively connected to adapter nodes configured to retrieve the data fragments from the respective data stores by identifying the data fragments from an object I

Assignees

• Palantir Technologies Inc

Inventors

• Wilczynski Peter
• Ryan Alexander
• Chang Allen
• Cervelli Daniel
• Tibshirani Julie
• Wilson Timothy

Classifications

• G06F16/256

  in federated or virtual databases · CPC title

• G06F16/2474

  Sequence data queries, e.g. querying versioned data · CPC title

• G06F16/2458 Primary

  Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries · CPC title

• G06F16/219 Primary

  Managing data history or versioning (querying versioned data G06F16/2474; querying temporal data G06F16/2477) · CPC title

• G06F16/2456

  Join operations · CPC title