Enforcing data security constraints in a data pipeline

Patent metadata

Field	Value
Publication number	US-12079352-B2
Application number	US-202117226014-A
Country	US
Kind code	B2
Filing date	Apr 8, 2021
Priority date	Dec 18, 2020
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

A computer-implemented method enforces data security constraints in a data pipeline. The data pipeline takes one or more source datasets as input and performs one or more data transformations on them. The method includes using data defining one or more data security constraints to configure the data pipeline to perform a data transformation on a restricted subset of entries of the source datasets. The restriction is defined by the data defining one or more data security constraints. The method further includes performing the data transformation according to the configuration to produce one or more transformed datasets. The method further includes using the data defining one or more data security constraints to perform a verification on one or more of the transformed datasets to ensure that entries in the one or more of the transformed datasets are restricted as defined by the one or more data security constraints.

First claim

What is claimed is:

1. A computer-implemented method for enforcing data security constraints in a data pipeline, wherein the data pipeline takes one or more source datasets as input and performs one or more data transformations on them, the method comprising:
within a first stage of the data pipeline, generate a first transformed dataset by performing a first data transformation on a first subset of entries of the one or more source datasets, wherein the first subset is defined according to one or more first data security constraints, wherein the one or more first data security constraints are associated with one or more columns or rows, and wherein an entry is accepted into or rejected from the first transformed dataset based on a comparison between the entry and the one or more first data security constraints;
within a second stage of the data pipeline, generate a second transformed dataset by performing a second data transformation on a second subset of entries of the one or more source datasets;
validate the second transformed dataset according to a pattern or constraint specified by the first transformed dataset, wherein the validating comprises comparing entries of the second transformed dataset against the first transformed dataset and filtering out any fields of the second transformed dataset that fail to conform to the pattern or constraint specified by the first transformed dataset, the first transformed dataset specifying a previously unknown or undefined criteria; and
providing an alert if any fields of the second transformed dataset fail to conform to the pattern.

2. The method of claim 1 , further comprising, prior to the first stage:
obtaining, from a user, data defining the first data security constraints to be applied to the first data transformation.

3. The method of claim 1 , wherein the one or more first data security constraints defines one or more conditions based on which the entry or a different entry in the one or more source datasets is either accepted or rejected for inclusion in the first subset of entries according to the one or more first data security constraints.

4. The method of claim 3 , wherein the one or more first data security constraints defines one or more acceptable values for entries of a certain type, and wherein the entry is accepted or rejected based on whether the entry matches the one or more acceptable values.

5. The method of claim 1 , wherein the first data transformation is a pre-existent data transformation of the data pipeline.

6. The method of claim 1 , wherein the second subset of entries of the one or more source datasets is defined according to one or more second data security constraints; and the validating of the second transformed dataset is based on the one or more second data security constraints.

7. The method of claim 6 , further comprising validating the first transformed dataset; and in response to the first transformed dataset being successfully validated, refraining from using the first data security constraints to perform the validation on the second transformed dataset.

8. The method of claim 7 , further comprising:
communicating the second transformed dataset to an external entity if the validation of the second transformed dataset is successful.

9. The method of claim 1 , wherein the second transformed dataset is released if the validation of the second transformed dataset is successful.

10. The method of claim 1 , further comprising:
preventing communication of the second transformed dataset if the validation fails.

11. The method of claim 1 , further comprising:
communicating the second transformed dataset to an external entity if the validation succeeds.

12. The method of claim 1 , wherein the second subset of entries reference at least a portion of the first subset of entries as a foreign key.

13. The method of claim 1 , further comprising receiving an update to the first transformed dataset; and
validating the second transformed dataset or a third transformed dataset according to a second pattern of the updated first transformed dataset.

14. A data processing system configured to enforce data security constraints in a data pipeline, wherein the data pipeline takes one or more source datasets as input and performs one or more data transformations on them, the data processing system including one or more processors and instructions that, when executed by the one or more processors, cause the data processing system to perform:
within a first stage of the data pipeline, generate a first transformed dataset by performing a first data transformation on a first subset of entries of the one or more source datasets, wherein the first subset is defined according to one or more first data security constraints, wherein the one or more first data security constraints are associated with one or more columns or rows, and wherein an entry is accepted into or rejected from the first transformed dataset based on a comparison between the entry and the one or more first data security constraints;
within a second stage of the data pipeline, generate a second transformed dataset by performing a second data transformation on a second subset of entries of the one or more source datasets;
validate the second transformed dataset according to a pattern or constraint specified by the first transformed dataset, wherein the validating comprises comparing entries of the second transformed dataset against the first transformed dataset and filtering out any fields of the second transformed dataset that fail to conform to the pattern or constraint specified by the first transformed dataset, the first transformed dataset specifying a previously unknown or undefined criteria; and
providing an alert if any fields of the second transformed dataset fail to conform to the pattern.

15. The data processing system of claim 14 , wherein the instructions further cause the data processing system to perform:
prior to the first stage:
obtaining, from a user, data defining the first data security constraints to be applied to the first data transformation.

16. The data processing system of claim 14 , wherein the one or more first data security constraints defines one or more conditions based on which the entry or a different entry in the one or more source datasets is either accepted or rejected for inclusion in the first subset of entries according to the one or more first data security constraints.

17. The data processing system of claim 16 , wherein the one or more first data security constraints defines one or more acceptable values for entries of a certain type, and wherein the entry is accepted or rejected based on whether the entry matches the one or more acceptable values.

18. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
within a first stage of the data pipeline, generate a first transformed dataset by performing a first data transformation on a first subset of entries of the one or more source datasets, wherein the first subset is defined according to one or more first data security constraints, wherein the one or more first data security constraints are associated with one or more columns or rows, and wherein an entry is accepted into or rejected from the first transformed dataset based on a comparison between the entry and the one or more first data security constraints;
within a second stage of the data pipeline, generate a second transformed dataset by performing a second data transformation on a second subset of entries of t

Assignees

• Palantir Technologies Inc

Inventors

• Apostolatos Anton
• Lieskovský Adam
• Diegruber Florian
• Ferreira Francisco
• Kane Joseph
• Peller Joanna
• Lau Kelvin
• Laska Maciej
• Mofarrej Mikael Ibrahim
• Schrader Max-Philipp
• Hoefer Philipp
• Mccollester Spencer
• Nordling Viktor

Classifications

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F21/6227

  where protection concerns the structure of data, e.g. records, types, queries · CPC title

• G06F21/604 Primary

  Tools and structures for managing or administering access control systems · CPC title

• G06F21/6245 Primary

  Protecting personal data, e.g. for financial or medical purposes · CPC title