Optimizing execution of data transformation flows

Patent metadata

Field	Value
Publication number	US-10242079-B2
Application number	US-201715701381-A
Country	US
Kind code	B2
Filing date	Sep 11, 2017
Priority date	Nov 7, 2016
Publication date	Mar 26, 2019
Grant date	Mar 26, 2019

Abstract

A computer system transforms data. The system displays a user interface including a data flow pane. A user builds a flow diagram in the data flow pane. Each node in the flow diagram specifies an operation: to retrieve data, to transform data, or to create an output dataset. The flow diagram includes a subtree having a data source node and transformation operation nodes. When the user initiates execution and the nodes in the subtree are configured to execute imperatively, the system performs the operations in the subtree sequentially as specified, retrieving data from the data source, transforming the data, and forming an intermediate dataset. When the user initiates execution and the nodes in the subtree are configured to execute declaratively, the system constructs a database query that is logically equivalent to the operations specified in the subtree and transmits the query to the data source to retrieve the intermediate dataset.

First claim

What is claimed is:

1. A computer system for transforming data, comprising:
one or more processors;
memory; and
one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs comprising instructions for:
displaying a user interface that includes a data flow pane and a data pane;
receiving first user input to build a node/link data transformation flow diagram in the data flow pane, wherein each node in the flow diagram specifies a respective operation to retrieve data from a respective data source, specifies a respective operation to transform data, or specifies a respective operation to create a respective output data set, and wherein the flow diagram includes a subtree having one or more data source nodes that retrieve data from a first data source and one or more transformation operation nodes;
receiving a second user input to execute at least the subtree;
in accordance with the second user input and a determination that the nodes in the subtree are configured to execute imperatively:
performing the operations of the nodes in the subtree sequentially as specified by links in the subtree, thereby retrieving data from the first data source, transforming the retrieved data, and forming a first intermediate data set; and
displaying the first intermediate data set in the data pane;

receiving third user input to configure the nodes in the subtree to execute declaratively;
receiving a fourth user input to execute at least the subtree; and
in accordance with the fourth user input and a determination that the nodes in the subtree are configured to execute declaratively:
constructing a database query that is logically equivalent to the operations specified by the nodes in the subtree;
transmitting the database query to the first data source to retrieve a second intermediate data set from the first data source according to the database query; and
displaying the second intermediate data set in the data pane.

2. The computer system of claim 1 , wherein the one or more programs further comprise instructions for storing the first intermediate data set and the second intermediate data set.

3. The computer system of claim 1 , wherein the subtree is the entire flow diagram.

4. The computer system of claim 1 , wherein:
one of the transformation operation nodes specifies a filter transformation operation to filter rows of data received by the one node;
performing the operations of the nodes in the subtree sequentially includes performing the filter transformation operation at the computer system to filter out received rows of data; and
retrieving the second intermediate data set from the first data source according to the database query includes applying the filter operation at a remote server hosting the first data source.

5. The computer system of claim 1 , wherein:
one of the transformation operation nodes specifies a join transformation operation that joins two sets of data from the first data source;
performing the operations of the nodes in the subtree sequentially includes performing the join transformation operation to combine the two sets of data at the computer system; and
retrieving the second intermediate data set from the first data source according to the database query includes applying the join operation at a remote server hosting the first data source.

6. The computer system of claim 1 , wherein the database query is written in SQL.

7. The computer system of claim 1 , wherein:
the flow diagram includes a portion not included in the subtree;
the portion is configured to execute imperatively;
the fourth user input specifies execution of the entire flow diagram; and
executing the flow diagram includes performing the operations of the nodes in the portion sequentially as specified by links in the portion, thereby accessing the second intermediate data set, transforming the second intermediate data set, and forming a final data set.

8. A method of transforming data, comprising:
at a computer system having a display, one or more processors, and memory storing one or more programs configured for execution by the one or more processors:
displaying a user interface that includes a data flow pane and a data pane;
receiving first user input to build a node/link data transformation flow diagram in the data flow pane, wherein each node in the flow diagram specifies a respective operation to retrieve data from a respective data source, specifies a respective operation to transform data, or specifies a respective operation to create a respective output data set, and wherein the flow diagram includes a subtree having one or more data source nodes that retrieve data from a first data source and one or more transformation operation nodes;
receiving a second user input to execute at least the subtree;
in accordance with the second user input and a determination that the nodes in the subtree are configured to execute imperatively:
performing the operations of the nodes in the subtree sequentially as specified by links in the subtree, thereby retrieving data from the first data source, transforming the retrieved data, and forming a first intermediate data set; and
displaying the first intermediate data set in the data pane;

receiving third user input to configure the nodes in the subtree to execute declaratively;
receiving a fourth user input to execute at least the subtree; and
in accordance with the fourth user input and a determination that the nodes in the subtree are configured to execute declaratively:
constructing a database query that is logically equivalent to the operations specified by the nodes in the subtree;
transmitting the database query to the first data source to retrieve a second intermediate data set from the first data source according to the database query; and
displaying the second intermediate data set in the data pane.

9. The method of claim 8 , further comprising storing the first intermediate data set and the second intermediate data set.

10. The method of claim 8 , wherein the subtree is the entire flow diagram.

11. The method of claim 8 , wherein:
one of the transformation operation nodes specifies a filter transformation operation to filter rows of data received by the one node;
performing the operations of the nodes in the subtree sequentially includes performing the filter transformation operation at the computer system to filter out received rows of data; and
retrieving the second intermediate data set from the first data source according to the database query includes applying the filter operation at a remote server hosting the first data source.

12. The method of claim 8 , wherein:
one of the transformation operation nodes specifies a join transformation operation that joins two sets of data from the first data source;
performing the operations of the nodes in the subtree sequentially includes performing the join transformation operation to combine the two sets of data at the computer system; and
retrieving the second intermediate data set from the first data source according to the database query includes applying the join operation at a remote server hosting the first data source.

13. The method of claim 8 , wherein the database query is written in SQL.

14. The method of claim 8 , wherein:
the flow diagram includes a portion not included in the subtree;
the portion is configured to execute imperatively;
the fourth user input specifies execution of the entire flow diagram; and
executing the flow diagram includes performing the operations of the nodes in the portion sequentially as specif

Assignees

• Tableau Software Inc

Inventors

• Kim Jun
• Pugh Will
• Kunen Isaac

Classifications

• G06F16/2455

  Query execution · CPC title

• G06F16/248

  Presentation of query results · CPC title

• G06F16/2453

  Query optimisation · CPC title

• G06F16/258 Primary

  Data format conversion from or to a database · CPC title

• G06F17/30477

  Physics · mapped topic