Extensible data transformation authoring and validation system

Patent metadata

Field	Value
Publication number	US-10860299-B2
Application number	US-201916384691-A
Country	US
Kind code	B2
Filing date	Apr 15, 2019
Priority date	Dec 13, 2016
Publication date	Dec 8, 2020
Grant date	Dec 8, 2020

Abstract

Data transformation in a distributed system of applications and data repositories is described. The subsystems for the overall framework are distributed, thereby allowing for customization to require only isolated changes to one or more subsystems. In one embodiment, a source code repository is used to receive and store source code. A build subsystem can retrieve source code from the source code repository and build it, using one or more criteria. By building the source code, the build subsystem can generate an artifact, which is executable code, such as a JAR or SQL file. Likewise, by building the source code, the build subsystem can generate one or more job specifications for executing the executable code. In one embodiment, the artifact and job specification may be used to launch an application server in a cluster. The application server can then receive data transformation instructions and execute the data transformation instructions.

First claim

What is claimed is:

1. A method, comprising:
receiving user code from a computer data storage device or input from a computer input device, the user code comprising a function that comprises a sequence of computation instructions for a dataset transformation that generates an output dataset, and a decorator that corresponds to the function that identifies one or more dependent datasets that are necessary to execute the function and one or more expected input datasets for a lower-order function that defines a data transformation job;
digitally storing the user code at a first code repository;
building the user code into executable code comprising one or more machine executable computer program files;
receiving a data transformation command that identifies the function;
in response to receiving the data transformation command:

based on the one or more expected input datasets for the lower-order function, using the executable code to invoke a higher-order function that corresponds to the function to identify and retrieve the one or more dependent datasets;
executing the function, using the one or more dependent datasets and the lower-order function, to generate a particular output dataset.

2. The method of claim 1 , further comprising, in response to receiving the data transformation command, using the decorator that corresponds to the function to generate a dataset dependency file that identifies the one or more dependent datasets.

3. The method of claim 1 , wherein the decorator further identifies one or more dataset types for the one or more dependent datasets that are necessary to execute the function.

4. The method of claim 1 , wherein the user code is written in a Python programming language and comprises at least a portion of a computer program.

5. The method of claim 4 , wherein the user code uses one or more of Python libraries comprising Pandas, NumPy, SciPy, or IPython Notebook.

6. The method of claim 1 , further comprising specifying, using the decorator, metaprogramming dependencies of the one or more dependent datasets that are necessary to execute the function.

7. The method of claim 1 , wherein the decorator further defines an expected location of input datasets, expected content of the input datasets or an expected characteristic of the input datasets, and wherein the one or more dependent datasets comprise the input datasets.

8. One or more non-transitory computer readable storage media storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform:
receiving user code from a computer data storage device or input from a computer input device, the user code comprising a function that comprises a sequence of computation instructions for a dataset transformation that generates an output dataset and a decorator that corresponds to the function that identifies one or more dependent datasets that are necessary to execute the function and one or more expected input datasets for a lower-order function that defines a data transformation job;
digitally storing the user code at a first code repository;
building the user code into executable code comprising one or more machine executable computer program files;
receiving a data transformation command that identifies the function;
in response to receiving the data transformation command:
based on the one or more expected input datasets for the lower-order function, using the executable code to invoke a higher-order function that corresponds to the function to identify and retrieve the one or more dependent datasets;
executing the function, using the one or more dependent datasets and the lower-order function, to generate a particular output dataset.

9. The one or more non-transitory computer readable media of claim 8 , further comprising sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform:
in response to receiving the data transformation command, using the decorator that corresponds to the function to generate a dataset dependency file that identifies the one or more dependent datasets.

10. The one or more non-transitory computer readable media of claim 8 , wherein the decorator further identifies one or more dataset types for the one or more dependent datasets that are necessary to execute the function.

11. The one or more non-transitory computer readable media of claim 8 , wherein the user code is written in a Python programming language and comprises at least a portion of a computer program.

12. The one or more non-transitory computer readable media of claim 11 , wherein the user code uses one or more of Python libraries comprising Pandas, NumPy, SciPy, or IPython Notebook.

13. The one or more non-transitory computer readable media of claim 8 , wherein the decorator specifies metaprogramming dependencies of the one or more dependent datasets that are necessary to execute the function.

14. The one or more non-transitory computer readable media of claim 8 , wherein the decorator further defines an expected location of input datasets, an expected type of the input datasets, expected content of the input datasets or an expected characteristic of the input datasets, and wherein the one or more dependent datasets comprise the input datasets.

15. A computer system comprising:
a processor; and
a memory coupled to the processor and storing one or more sequences of instructions which, when executed by the processor, cause the processor to perform:
receiving user code from a computer data storage device or input from a computer input device, the user code comprising a function that comprises a sequence of computation instructions for a dataset transformation that generates an output dataset and a decorator that corresponds to the function that identifies one or more dependent datasets that are necessary to execute the function and one or more expected input datasets for a lower-order function that defines a data transformation job;
digitally storing the user code at a first code repository;
building the user code into executable code comprising one or more machine executable computer program files;
receiving a data transformation command that identifies the function;
in response to receiving the data transformation command:
based on the one or more expected input datasets for the lower-order function, using the executable code to invoke a higher-order function that corresponds to the function to identify and retrieve the one or more dependent datasets;
executing the function, using the one or more dependent datasets and the lower-order function, to generate a particular output dataset.

16. The system of claim 15 , wherein the user code is written in a Python programming language and comprises at least a portion of a computer program.

Assignees

• Palantir Technologies Inc

Inventors

• Fink Robert
• Cheah Matthew
• Kim Mingyu
• Cuthriell Lynn
• Arora Divyanshu
• Uang Justin
• Newman Jared
• Juelich Jakob
• Chen Kevin
• Elliot Mark
• Nazario Michael

Classifications

• G06F9/4488

  Object-oriented · CPC title

• G06F8/315 Primary

  Object-oriented languages · CPC title

• G06F8/71

  Version control (security arrangements therefor G06F21/57); Configuration management · CPC title

• G06F8/311

  Functional or applicative languages; Rewrite languages · CPC title

• G06F8/40 Primary

  Transformation of program code · CPC title