Automated syncing of data between security domains

Patent metadata

Field	Value
Publication number	US-11533292-B2
Application number	US-202016992459-A
Country	US
Kind code	B2
Filing date	Aug 13, 2020
Priority date	Jan 7, 2020
Publication date	Dec 20, 2022
Grant date	Dec 20, 2022

Abstract

Described herein are systems, methods, and non-transitory computer readable media for automating the transfer/syncing of datasets or other artifacts from one security domain (e.g., a low security side environment) to another security domain (e.g., a high security side environment) in a seamless manner that complies with requirements of a data transfer mechanism used to transfer data between the two security domains while ensuring data integrity and consistency between the two security domains.

First claim

What is claimed is:

1. A system, comprising:
at least one processor; and
at least one memory storing computer-executable instructions, wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
identify a dataset to be synced between a first security domain and a second security domain;
process the dataset, wherein the processing comprises partitioning a file in the dataset into a plurality of file segments, each of the plurality of file segments being smaller than a threshold file size permitted for transferring data between the first security domain and the second security domain;
store information of the processing in a command log;
transfer contents of the dataset from the first security domain to the second security domain, wherein the transferring of the contents of the dataset comprises:
transferring the plurality of file segments from the first security domain to the second security domain;

transfer metadata associated with the dataset from the first security domain to the second security domain, wherein the metadata comprises the command log, and a file path of a corresponding dataset in the second security domain;
combine the plurality of file segments in the second security domain to reconstruct the file based on the command log identifying the dataset to be synced; and
utilize the metadata to sync the corresponding dataset in the second security domain with the file from the first security domain based on the file path of the corresponding dataset, the file path identifying the contents of the dataset to be synced, wherein the syncing comprises an incremental sync between the contents of the data from the first security domain and the corresponding dataset in the second security domain.

2. The system of claim 1 , wherein the at least one processor is further configured to execute the computer-executable instructions to:
transfer the plurality of file segments from the first security domain to an output store; and
transfer the metadata from the first security domain to the output store.

3. The system of claim 2 , wherein the output store is located in the first security domain, and wherein the plurality of file segments and the metadata are transferred from the output store to the second security domain.

4. The system of claim 2 , wherein the output store is located in the second security domain, and wherein transfer of the plurality of file segments and the metadata to the second security domain is achieved by transfer of the plurality of file segments and the metadata to the output store.

5. The system of claim 1 , wherein the at least one processor is further configured to execute the computer-executable instructions to:
generate the dataset in the first security domain by deconstructing data such that the transfer of the plurality of file segments satisfies one or more data transfer limitations associated with a data transfer mechanism used to transfer the plurality of file segments from the first security domain to the second security domain; and reconstruct the dataset in the second security domain based at least in part on the metadata.

6. The system of claim 5 , wherein the one or more data transfer limitations comprise one or more of a restriction on the file types that can be transferred from the first security domain to the second security domain, a limit on the cumulative size of files that can be in-flight from the first security domain to the second security domain at the same time, or a limit on the number of API calls that can be made.

7. The system of claim 1 , wherein the plurality of file segments comprise one or more of an object view, an ontology, or code.

8. The system of claim 1 , wherein a first data stack of the first security domain is a subset of a second data stack of the second security domain.

9. The system of claim 1 , wherein the at least one processor is further configured to execute the computer-executable instructions to:
ingest, at the second security domain, the command log file;
apply a transform to the ingested command log file;
generate, based at least in part on the applied transform, a data ingest comprising the plurality of file segments to be synced; and
trigger the data ingest to sync the corresponding dataset in the second security domain with the file from the first security domain.

10. The system of claim 5 , wherein the data transfer mechanism is a one-way transfer device that only permits data to flow from the first security domain to the second security domain.

11. A computer-implemented method for automated syncing of data between security domains, the method comprising:
identifying a dataset to be synced between a first security domain and a second security domain;
processing the dataset, wherein the processing comprises partitioning a file in the dataset into a plurality of file segments, each of the plurality of file segments being smaller than a threshold file size permitted for transferring data between the first security domain and the second security domain;
storing information of the processing in a command log;
transferring contents of the dataset from the first security domain to the second security domain, wherein the transferring of the contents of the dataset comprises:
transferring the plurality of file segments from the first security domain to the second security domain;

transferring metadata associated with the dataset from the first security domain to the second security domain, wherein the metadata comprises the command log and a file path of a corresponding dataset in the second security domain;
combining the plurality of file segments in the second security domain to reconstruct the file based on the command log identifying the dataset to be synced; and
utilizing the metadata to sync the corresponding dataset in the second security domain with the file from the first security domain based on the file path of the corresponding dataset, the file path identifying the contents of the dataset to be synced, wherein the syncing comprises an incremental sync between the contents of the data from the first security domain and the corresponding dataset in the second security domain.

12. The computer-implemented method of claim 11 , further comprising:
transferring the plurality of file segments from the first security domain to an output store; and
transferring the metadata from the first security domain to the output store.

13. The computer-implemented method of claim 12 , wherein the output store is located in the first security domain, and wherein the plurality of file segments and the metadata are transferred from the output store to the second security domain.

14. The computer-implemented method of claim 12 , wherein the output store is located in the second security domain, and wherein transfer of the plurality of file segments and the metadata to the second security domain is achieved by transfer of the plurality of file segments and the metadata to the output store.

15. The computer-implemented method of claim 11 , further comprising:
deconstructing data such that the transfer of the plurality of file segments satisfies one or more data transfer limitations associated with a data transfer mechanism used to transfer the plurality of file segments from the first security domain to the second security domain; and
reconstructing the dataset in the second security domain based at least in part on the metadata.

16. The computer-implemented method of claim 15 , wherein the one or more data transfer limitations c

Assignees

• Palantir Technologies Inc

Inventors

• Ochs-Willard Jess
• Brady Matthew
• Belardo Armando
• Skiles Mitchell

Classifications

• H04L63/0428

  wherein the data content is protected, e.g. by encrypting or encapsulating the payload · CPC title

• G06F16/164

  File meta data generation · CPC title

• H04L63/0209 Primary

  Architectural arrangements, e.g. perimeter networks or demilitarized zones · CPC title

• H04L67/06

  specially adapted for file transfer, e.g. file transfer protocol [FTP] · CPC title

• G06F16/178

  Techniques for file synchronisation in file systems · CPC title