Dynamic streaming data dispatcher

Patent metadata

Field	Value
Publication number	US-9367501-B2
Application number	US-201113293822-A
Country	US
Kind code	B2
Filing date	Nov 10, 2011
Priority date	Nov 10, 2011
Publication date	Jun 14, 2016
Grant date	Jun 14, 2016

Abstract

A method includes receiving, by a computing device, a plurality of data streams from plurality of sources, distributing the data streams to a plurality of sinks on multiple hosts, receiving load information indicating a load on at least one of the plurality of sinks and adjusting the distribution of the data stream accordingly and instructing the plurality of sinks to write the data streams to a distributed data store.

First claim

What is claimed is:

1. A method, comprising:
receiving, by a computing device, a data stream from a source to be stored on a distributed data store;
distributing the data stream to a plurality of sinks on multiple hosts by sending a plurality of data tuples to the plurality of sinks, wherein each of the sinks on the multiple hosts are configured to buffer the data stream and wherein the distributed data store is configured to receive data from each of the multiple hosts, and wherein distributing the data stream to the plurality of sinks on multiple hosts comprises sending a data tuple from the plurality of data tuples to two or more of the plurality of sinks;
receiving load information indicating a load on at least one of the plurality of sinks and responsively adjusting the distribution of the data stream;
sending, by the computing device, to a first sink from the plurality of sinks, a control tuple, the control tuple comprising one of a write request or a discard request, the write request instructing the first sink to write data stored on the first sink to the distributed data store based on a buffer size and a load on the first sink, and the discard request instructing the first sink to discard the data stored on the first sink;
receiving a result tuple from the first sink indicating a success or failure of writing the data stream to the distributed data store; and
sending the control tuple to a second sink from the plurality of sinks in response to the result tuple from the first sink indicating a failure of writing the data stream to the distributed data store by the first sink.

2. The method of claim 1 , wherein a data rate of the data stream is variable.

3. The method of claim 1 , wherein distributing the data stream to a plurality of sinks on multiple hosts is performed in a balanced manner.

4. The method of claim 1 , wherein writing the data stream to the distributed store comprises simultaneously writing a plurality of buffered data tuples on the plurality of sinks to the distributed data store.

5. The method of claim 1 , wherein the data stream is distributed to the plurality of sinks based on a total size of data being sent to each sink.

6. A method, comprising:
receiving, by a computing device, a plurality of data streams from a plurality of data sources to be stored on a distributed data store;
distributing the plurality data stream to a plurality of sinks on multiple hosts by sending a plurality of data tuples to the plurality of sinks, wherein each of the sinks on the multiple hosts are configured to buffer the data stream and wherein the distributed data store is configured to receive data from each of the multiple hosts, wherein distributing the data stream to the plurality of sinks on multiple hosts comprises sending a data tuple from the plurality of data tuples to two or more of the plurality of sinks;
receiving load information indicating a load on at least one of the plurality of sinks and responsively adjusting the distribution of the data stream;
sending, by the computing device, to a first sink from the plurality of sinks, a control tuple, the control tuple comprising one of a write request or a discard request, the write request instructing the first sink to write data stored on the first sink to the distributed data store based on a buffer size and a load on the first sink, and the discard request instructing the first sink to discard the data stored on the first sink;
receiving a result tuple from the first sink indicating a success or failure of writing the data stream to the distributed data store in response to the control tuple; and
sending the control tuple to a second sink from the plurality of sinks in response to the result tuple from the first sink indicating a failure of writing the data stream to the distributed data store by the first sink.

7. The method of claim 6 , where a data rate of at least one of the plurality of data streams is variable.

8. The method of claim 6 , wherein writing the plurality of data streams to the distributed store comprises simultaneously writing a plurality of buffered data tuples on the plurality of sinks to the distributed data store.

9. A computer program product for distributing incoming data streams, the computer program product comprising:
a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code configured to:
receive, by a computing device a plurality of data streams from a plurality of data sources to be stored on distributed data store;
distribute the plurality data stream to a plurality of sinks on multiple hosts by sending a plurality of data tuples to the plurality of sinks, wherein each of the sinks on the multiple hosts are configured to buffer the data stream and wherein the distributed data store is configured to receive data from each of the multiple hosts, wherein distributing the data stream to the plurality of sinks on multiple hosts comprises sending a data tuple from the plurality of data tuples to two or more of the plurality of sinks;

receive load information indicating a load on at least one of the plurality of sinks and responsively adjust the distribution of the data stream;
send, by the computing device, to a first sink from the plurality of sinks, a control tuple, the control tuple comprising one of a write request or a discard request, the write request indicative for the first sink to write data stored on the first sink to the distributed data store based on a buffer size and a load on the first sink, and the discard request indicative for the first sink to discard the data stored on the first sink;
receive a result tuple from the first sink indicating a success or failure of writing the data stream to the distributed data store in response to the control tuple; and

send the control tuple to a second sink from the plurality of sinks in response to the result tuple from the first sink indicating a failure of writing the data stream to the distributed data store by the first sink.

10. The computer program product of claim 9 , wherein distributing the plurality of data streams to a plurality of sinks on multiple hosts is performed in a balanced manner.

11. A system, comprising:
a splitter for receiving a data stream from a data source;
a plurality of sinks in operable communication with the splitter;
a distributed data storage device in operable communication with the plurality of sinks; and
a load manager in operable communication with the plurality of sinks and the splitter, wherein the load manager instructs the splitter on distribution of the data stream to the plurality of sinks and instructs the plurality of sinks to write data stored on the plurality of sinks to the distributed data store based on a buffer size and a load on the plurality of the sinks; and wherein the splitter is configured to:
split the data stream into a plurality portions;
send a first data tuple from comprising a portion of the data stream to a first sink and a second sink, the first sink and the second sink being from the plurality of sinks;
send a control tuple to the first sink, the control tuple comprising a write request for the first sink to write the portion of the data stream in the data tuple to the distributed data store;
receive a result tuple from the first sink indicating a success or failure of writing the portion of the data stream to the distributed data store; and
send the control tuple to a second sink from the plurality of sinks in response to the result tuple from the first sink indicating a failure of writing the data stream to the distributed data s

Assignees

• Idziorek Joseph R
• Kumar Vibhore
• Venkatramani Chitra
• Wagle Rohit
• IBM

Inventors

• Idziorek Joseph R
• Kumar Vibhore
• Venkatramani Chitra
• Wagle Rohit

Classifications

• H04L47/125

  by balancing the load, e.g. traffic engineering · CPC title

• G06F13/385 Primary

  for adaptation of a particular data processing system to different peripheral devices · CPC title