Systems and methods for validating data

Patent metadata

Field	Value
Publication number	US-11221898-B2
Application number	US-201916675056-A
Country	US
Kind code	B2
Filing date	Nov 5, 2019
Priority date	Apr 10, 2017
Publication date	Jan 11, 2022
Grant date	Jan 11, 2022

Abstract

Systems and methods are validating data in a data set. A data set including data to validate and a validator to use in validating the data is selected based on user input generated based on interactions of a user with a graphical user interface. The validator is applied to the data to determine whether one or more statistics generated through application of the validator to the data is valid or invalid based on a validation routine associated with the validator. A data quality report indicating whether the data set is valid or invalid, based on a determination of whether the one or more statistics is valid or invalid, is generated and selectively presented to the user through the graphical user interface.

First claim

The invention claimed is:

1. A system for validating data comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to:
select a data set to be validated based on a user input of a user;
select, based on the user input, a first validator and a second validator with which to validate data in the data set, wherein the first validator comprises a first validation parameter and a first validation routine and the second validator comprises a different validation parameter or a different validation routine compared to the first validator;
generate, based on the first validation parameter and a validation parameter corresponding to the second validator, one or more statistics associated with the data set;
determine, based on the first validation routine and a validation routine corresponding to the second validator the second validation routine a validity of the data, wherein the validity of the data is determined based on whether the one or more statistics associated with the data set satisfies one or more conditions specified in the first validation routine and the validation routine corresponding to the second validator;
generate a data quality report for the data set indicating the validity of the data based on a determination that the one or more statistics associated with the data are valid; and
provide the data quality report to the user.

2. The system of claim 1 , wherein the system is further caused to select the first validation parameter based on the user input.

3. The system of claim 1 , wherein the system is further caused to:
select the first validation routine based on the user input;
customize the first validator by associating the first validator with the first validation routine selected based on the user input for use in applying the first validator to the data in the data set to determine whether the one or more statistics are valid according to the first validation routine.

4. The system of claim 1 , wherein the system is further caused to:
select the first validation parameter based on the user input;
customize the first validator by associating the first validation parameter with the first validator according to the user input.

5. The system of claim 1 , wherein the system is further caused to:
receive additional user input;
modify the first validator to create a modified validator by modifying the first validation parameter to create a modified validation parameter in the modified validator according to the additional user input;
apply the modified validator to the data in the data set to determine whether an additional one or more statistics generated through application of the modified validator to the data in the data set using the modified validation parameter are valid according to the first validation routine;
generate a second data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the additional one or more statistics are valid according to the first validation routine; and
selectively present the second data quality report to the user through the graphical user interface.

6. The system of claim 1 , wherein the system is further caused to:
receive additional user input;
customize the first validator to create a customized validator by modifying the first validation routine to create a modified validation routine in the customized validator according to the additional user input;
apply the customized validator to the data in the data set to determine whether an additional one or more statistics generated through application of the customized validator to the data in the data set using the first validation parameter are valid according to the modified validation routine;
generate a second data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the additional one or more statistics are valid according to the modified validation routine; and
selectively present the second data quality report to the user through the graphical user interface.

7. The system of claim 1 , wherein the system is further caused to:
re-apply the first validator to the data in the data set to determine whether the one or more statistics generate through application of the first validator to the data in the data set using the first validation parameter are valid according to the first validation routine;
generate a second data quality report for the data indicating whether the data set is invalid or invalid based on a determination made through re-application of the first validator to the data in the data set of whether the one or more statistics are valid according to the first validation routine; and
selectively present the second data quality report to the user through the graphical user interface.

8. The system of claim 7 , wherein the system is further caused to re-apply the first validator to the data in the data set based on one or a combination of re-application instructions received from a user, a passage of a specific amount of time, and a change made to the data in the data set.

9. The system of claim 1 , wherein the system is further caused to:
select a second data set including additional data to validate using the first validator based on the user input;
concurrently with application of the first validator and the second validator to the data in the data set, apply the first validator to the additional data in the another data set to determine whether one or more additional statistics generated through application of the first validator to the additional data in the another data set using the first validation parameter is valid according to the first validation routine associated with the first validator;
generate a second data quality report for the data set and the second data set indicating whether either or both the data set and the second data set are valid or invalid based on a determination of whether either or both the one or more statistics and the one or more additional statistics are valid according to the first validation routine; and
selectively present the second data quality report to the user through the graphical user interface.

10. The system of claim 1 , wherein the first validation parameter indicates a particular location within the data set to generate the one or more statistics.

11. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
selecting a data set to be validated based on a user input of a user;
selecting, based on the user input, a first validator and a second validator with which to validate data in the data set, wherein the first validator comprises a first validation parameter and a first validation routine and the second validator comprises a different validation parameter or a different validation routine compared to the first validator;
generating, based on the first validation parameter and a validation parameter corresponding to the second validator, one or more statistics associated with the data set;
determining, based on the first validation routine and a validation routine corresponding to the second validator, a validity of the data, wherein the validity of the data is determined based on whether the one or more statistics associated with the data set satisfies one or more conditions specified in the first validation routine and the validation routine corresponding to the second validator;
generating a data quality report for the data set indicating the validity of the data b

Assignees

• Palantir Technologies Inc

Inventors

• Lisuk David
• Xu Guodong
• Voloch Luis
• Elkherj Matthew

Classifications

• G06F16/43

  Querying · CPC title

• G06F16/23

  Updating · CPC title

• G06F11/0709 Primary

  in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06Q10/10 Primary

  Office automation; Time management · CPC title