Approaches for managing object data

Patent metadata

Field	Value
Publication number	US-11789904-B2
Application number	US-202217948554-A
Country	US
Kind code	B2
Filing date	Sep 20, 2022
Priority date	Oct 11, 2019
Publication date	Oct 17, 2023
Grant date	Oct 17, 2023

Abstract

Systems and methods are provided for determining multiple fragments of data to be imported, the multiple fragments of data corresponding to different instances of data obtained from one or more external data sources, the different instances of data each corresponding to duplicate content. The multiple fragments of data that each correspond to different instances of duplicate content can be ingested. The multiple fragments of data can be de-duplicated to determine one or more corresponding object data source records (DSRs). The one or more object DSRs can be imported within a data platform system.

First claim

The invention claimed is:

1. A system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
determining fragments of data to be imported, the fragments of data corresponding to different instances of data obtained from one or more external data sources;
ingesting the fragments of data to a data platform;
determining one or more respective object data source records (DSRs) of the fragments of data, wherein the one or more respective object DSRs correspond to data identifiers;
importing the one or more object DSRs within the data platform;
determining that access control information associated with a first fragment of the fragments has been modified into modified access control information, wherein the first fragment is associated with a particular object DSR and corresponds to a particular data identifier; and
selectively creating a new object DSR depending on a presence or an absence of a second fragment associated with the modified access control information and the particular data identifier.

2. The system of claim 1 , wherein the selective modification of the particular object DSR is based on a determination of whether a second fragment is associated with the access control information and the particular object DSR.

3. The system of claim 1 , wherein the selective modification of the particular object DSR comprises:
in response to determining that a second fragment is associated with the access control information and the particular object DSR, refraining from modifying the particular object DSR; and
in response to determining a nonexistence of any fragments that are associated with the access control information and the particular object DSR, modifying the particular object DSR.

4. The system of claim 1 , wherein the instructions further cause the system to perform:
receiving an indication that, in response to determining that the access control information associated with the first fragment has been modified into the modified access control information, the first fragment has been deleted;
in response to receiving the indication that the first fragment has been deleted, and in response to determining that any fragment within the data platform is associated with a different object DSR than the particular object DSR or with different access control information than the access control information, deleting the particular object DSR.

5. The system of claim 4 , wherein the instructions further cause the system to perform:
in response to determining that any fragment within the data platform is associated with a different object DSR than the particular object DSR or with different access control information than the access control information, determining second access control information of any fragment within the data platform that was previously associated with the particular object DSR and the access control information; and
modifying the particular object DSR according to the second access control information or the modified access control information.

6. The system of claim 5 , wherein the instructions further cause the system to perform:
in response to determining that no other fragment is being modified with respect to the access control information, modifying the particular object DSR according to the modified access control information.

7. The system of claim 5 , wherein the modifying of the particular object DSR according to the modified second access control information or the modified access control information comprises a deterministic process to select whether the modifying is according to the modified second access control information or the modified access control information.

8. The system of claim 1 , wherein the instructions further cause the system to perform:
hiding or deleting, in the data platform, the access control information associated with the first fragment in response to:
determining that the access control information associated with the first fragment has been modified into the modified access control information.

9. The system of claim 1 , wherein, in response to the modification of the access control information into the modified access control information, a hash value of the first fragment remains unchanged.

10. A computer-implemented method, comprising:
determining fragments of data to be imported, the fragments of data corresponding to different instances of data obtained from one or more external data sources;
ingesting the fragments of data to a data platform;
determining one or more respective object data source records (DSRs) of the fragments of data, wherein the one or more respective object DSRs correspond to data identifiers;
importing the one or more object DSRs within the data platform;
determining that first access control information associated with a first fragment of the fragments has been modified into modified access control information, wherein the first fragment is associated with a first particular object DSR;
determining that second access control information associated with a second fragment of the fragments has been modified into the modified access control information, wherein the second fragment is associated with a second particular object DSR;
selectively modifying the first particular object DSR and the second particular DSR based on the modified access control information; and
in response to receiving an indication that the first fragment or the second fragment has been deleted, selectively deleting the first particular object DSR or the second particular object DSR.

11. The computer-implemented method of claim 10 , wherein the selective modification of the first particular object DSR is based on a determination of whether a third fragment is associated with the modified access control information and the first particular object DSR.

12. The computer-implemented method of claim 10 , wherein the selective modification of the first particular object DSR comprises:
in response to determining that a third fragment is associated with the modified access control information and the first particular object DSR, refraining from modifying the first particular object DSR; and
in response to determining a nonexistence of any fragments that are associated with the modified access control information and the first particular object DSR, modifying the first particular object DSR.

13. The computer-implemented method of claim 11 , further comprising:
in response to determining that no other fragment is being modified with respect to the first access control information, modifying the first particular object DSR according to the modified access control information.

14. The computer-implemented method of claim 10 , wherein the selective deleting the first particular object DSR or the second particular object DSR comprises:
selectively hiding or deleting, in the data platform, the first access control information associated with the first fragment in response to determining that the first access control information associated with the first fragment has been modified into the modified access control information.

15. The computer-implemented method of claim 10 , wherein, in response to the modification of the first access control information into the modified access control information, a hash value of the first fragment remains unchanged.

16. A system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
determining fragm

Assignees

• Palantir Technologies Inc

Inventors

• Chang Allen
• Rochau Denis
• Raboin Eric
• Ossher Joel
• Stoll Sam

Classifications

• G06F16/152 Primary

  using file content signatures, e.g. hash values · CPC title

• G06F16/174

  Redundancy elimination performed by the file system (error detection or correction of the data by redundancy in operations G06F11/14) · CPC title

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title