Domain-specific language for dataset transformations

Patent metadata

Field	Value
Publication number	US-9965534-B2
Application number	US-201615369753-A
Country	US
Kind code	B2
Filing date	Dec 5, 2016
Priority date	Sep 9, 2015
Publication date	May 8, 2018
Grant date	May 8, 2018

Abstract

Techniques related to a domain-specific language for dataset transformations are disclosed. A server computer may process a table definition composed in a domain-specific language. The table definition may include a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table. The sequence may include a customized transformation. A source dataset may be provided as input to an implementation of the customized transformation. An output dataset may be generated as a result of executing the implementation. An intermediate table may be generated based on performing at least one dataset transformation on a particular source table. A supplemental portion for the intermediate table may be generated based on performing the at least one dataset transformation on an appended portion of the particular source table. The target table may be generated based on combining the supplemental portion with the intermediate table.

First claim

What is claimed is:

1. A data processing method comprising:
processing a table definition composed in a domain-specific language, the table definition comprising a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table;
retrieving an intermediate table that was generated based on performing a first dataset transformation of the one or more dataset transformations on a previous version of a particular source table of the one or more source tables;
generating a supplemental portion for the intermediate table based on performing a second dataset transformation on an appended portion of an updated version of the particular source table;
generating the target table based on performing a third dataset transformation on the intermediate table and the supplemental portion for the intermediate table;
wherein the method is performed by one or more computing devices.

2. The method of claim 1 , wherein the first dataset transformation and the second dataset transformation are different dataset transformations.

3. The method of claim 1 , wherein the second dataset transformation and the third dataset transformation are performed without being specified by an end user.

4. The method of claim 1 , wherein generating the supplemental portion comprises performing a fourth dataset transformation on an output dataset of the first dataset transformation and an output dataset of the second dataset transformation.

5. The method of claim 4 , wherein the fourth dataset transformation is reversible.

6. The method of claim 1 , wherein the target table is generated based on persisting an output dataset of the third dataset transformation.

7. The method of claim 1 , wherein the sequence of one or more dataset transformations is determined based on a graphical representation of the one or more dataset transformations.

8. The method of claim 7 , wherein the graphical representation is a directed acyclic graph.

9. The method of claim 7 , wherein the graphical representation comprises one or more leaf nodes and one or more non-leaf nodes, the one or more leaf nodes representing one or more tables, the one or more non-leaf nodes representing the one or more dataset transformations.

10. The method of claim 1 , wherein one or more references to the one or more source tables are stored at a server computer, without copies of the one or more source tables being stored at the server computer.

11. A computer system comprising:
one or more processors;
one or more non-transitory data storage media coupled to the one or more processors and storing instructions which, when executed by the one or more processors, cause:
processing a table definition composed in a domain-specific language, the table definition comprising a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table;
retrieving an intermediate table that was generated based on performing a first dataset transformation of the one or more dataset transformations on a previous version of a particular source table of the one or more source tables;
generating a supplemental portion for the intermediate table based on performing a second dataset transformation on an appended portion of an updated version of the particular source table;
generating the target table based on performing a third dataset transformation on the intermediate table and the supplemental portion for the intermediate table.

12. The system of claim 11 , wherein the first dataset transformation and the second dataset transformation are different dataset transformations.

13. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause performing the second dataset transformation and the third dataset transformation without being specified by an end user.

14. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause performing generating the supplemental portion by performing a fourth dataset transformation on an output dataset of the first dataset transformation and an output dataset of the second dataset transformation.

15. The system of claim 14 , wherein the fourth dataset transformation is reversible.

16. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause generating the target table based on persisting an output dataset of the third dataset transformation.

17. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause determining the sequence of one or more dataset transformations based on a graphical representation of the one or more dataset transformations.

18. The system of claim 17 , wherein the graphical representation is a directed acyclic graph.

19. The system of claim 17 , wherein the graphical representation comprises one or more leaf nodes and one or more non-leaf nodes, the one or more leaf nodes representing one or more tables, the one or more non-leaf nodes representing the one or more dataset transformations.

20. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause storing one or more references to the one or more source tables at a server computer, without corresponding copies of the one or more source tables being stored at the server computer.

Assignees

• Palantir Technologies Inc

Inventors

• Tolnay David
• Biswal Punyashloka
• Colombi Andrew
• Fu Yupeng
• Fuadi Ashar
• Kim Mingyu
• Nepywoda Paul
• Pundle Akshay
• Tamayo Juan

Classifications

• G06F16/23

  Updating · CPC title

• G06F17/30569 Primary

  Physics · mapped topic

• G06F17/30345

  Physics · mapped topic

• G06F17/30563

  Physics · mapped topic

• G06F17/30961

  Physics · mapped topic