Processing data from multiple sources

Patent metadata

Field	Value
Publication number	US-11403308-B2
Application number	US-202016865975-A
Country	US
Kind code	B2
Filing date	May 4, 2020
Priority date	Apr 17, 2014
Publication date	Aug 2, 2022
Grant date	Aug 2, 2022

Abstract

In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

First claim

What is claimed is:

1. A method including:
at a node of a cluster that stores a collection of data that can be operated on in parallel by nodes operating in conjunction with one another to carry out data processing operations on the collection of data, the node storing a first portion of data of the collection of data, the first portion of data stored in HDFS data storage:
executing a first instance of a data processing engine capable of accessing the first portion of data stored in HDFS data storage and receiving data from a data source external to the cluster;
receiving a computer program by the data processing engine, the computer program configured for accessing the first portion of data stored in HDFS data storage and including a) at least one component representing the cluster, b) at least one component representing the data source external to the cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;
executing at least part of the computer program by the first instance of the data processing engine;
accessing, by the data processing engine, the first portion of data stored in HDFS data storage;
receiving, by the data processing engine, a second portion of data from the external data source; and
performing, by the data processing engine, the data processing operation using at least the first portion of data stored in HDFS data storage and the second portion of data from the external data source.

2. The method of claim 1 in which the nodes each execute an instance of the data processing engine, the instances of the data processing engine running concurrently to perform the data processing operation together in parallel.

3. The method of claim 2 in which the data processing operation is performed on a) a first body of data that includes the first portion of data, the first body of data also including other portions of data being processed by the other nodes of the cluster, and b) a second body of data that includes the second portion of data, the second body of data being stored in a format native to a relational database system, and the second body of data being divided into portions that each can be stored in volatile memory of the nodes of the cluster.

4. The method of claim 1 in which at least one component of the computer program is connected to a link representing a flow of data from the cluster, and in which the at least one component is connected to a link representing a flow of data from the source of the second portion of data.

5. The method of claim 1 in which at least some data of the collection of data is stored redundantly in multiple nodes.

6. The method of claim 1 in which the second portion of data is stored in volatile memory of the node.

7. The method of claim 1 including
receiving a database query, the database query including at least one operation to be performed on data received from at least one source of data that includes the cluster; and
the computer program including components representing operations corresponding to the database query, in which the computer program includes at least one component representing the at least one source of data and at least one link that represents at least one dataflow associated with the operation to be performed on data received from at least one source of data.

8. The method of claim 1 in which the second portion of data is chosen based on characteristics of the first portion of data.

9. The method of claim 1 in which the second portion of data includes a subset of rows of a relational database, and the second portion of data includes a subset of columns of the relational database.

10. The method of claim 1 in which the second portion of data is distinct from a third portion of data received from the external data source at another node of the cluster.

11. The method of claim 1 including communicating with an instance of at least part of the computer program that is being executed by a second instance of the data processing engine that is outside of the cluster.

12. The method of claim 1 including executing at least part of the computer program by a second instance of the data processing engine outside of the cluster.

13. A non-transitory computer-readable medium including instructions for causing a node of a cluster to carry out operations, the cluster storing a collection of data that can be operated on in parallel by nodes operating in conjunction with one another to carry out data processing operations on the collection of data, the node storing a first portion of data of the collection of data, the first portion of data stored in HDFS data storage, the operations including:
executing a first instance of a data processing engine capable of accessing the first portion of data stored in HDFS data storage and receiving data from a data source external to the cluster;
receiving a computer program by the data processing engine, the computer program configured for accessing the first portion of data stored in HDFS data storage and including a) at least one component representing the cluster, b) at least one component representing the data source external to the cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;
executing at least part of the computer program by the first instance of the data processing engine;
accessing, by the data processing engine, the first portion of data stored in HDFS data storage;
receiving, by the data processing engine, a second portion of data from the external data source; and
performing, by the data processing engine, the data processing operation using at least the first portion of data stored in HDFS data storage and the second portion of data from the external data source.

14. A node of a cluster that stores a collection of data that can be operated on in parallel by nodes operating in conjunction with one another to carry out data processing operations on the collection of data, the node storing a first portion of data of the collection of data, the first portion of data stored in HDFS data storage, the node including a computer processing device configured to carry out operations including:
executing a first instance of a data processing engine capable of accessing the first portion of data stored in HDFS data storage and receiving data from a data source external to the cluster;
receiving a computer program by the data processing engine, the computer program configured for accessing the first portion of data stored in HDFS data storage and including a) at least one component representing the cluster, b) at least one component representing the data source external to the cluster, and c) at least one link that represents at least one dataflow associated with a data processing operation;
executing at least part of the computer program by the first instance of the data processing engine;
accessing, by the data processing engine, the first portion of data stored in HDFS data storage;
receiving, by the data processing engine, a second portion of data from the external data source; and
performing, by the data processing engine, the data processing operation using at least the first portion of data stored in HDFS data storage and the second portion of data from the external data source.

15. A node of a cluster that stores a collection of data that can be operated on in parallel by nodes operating in conjunction with one another to carry out data processing operations on the collection of data, the node storing a first portion of data of the collection of data, the first porti

Assignees

• Ab Initio Technology Llc

Inventors

• Schechter Ian
• Wakeling Tim
• Wollrath Ann M

Classifications

• G06F16/2471 Primary

  Distributed queries · CPC title

• G06F9/5066 Primary

  Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451) · CPC title

• G06F16/13

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title

• G06F16/1734

  Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title