Collecting streaming event data from applications deployed in a distributed computing environment

Patent metadata

Field	Value
Publication number	US-9870272-B2
Application number	US-201414539049-A
Country	US
Kind code	B2
Filing date	Nov 12, 2014
Priority date	Nov 12, 2013
Publication date	Jan 16, 2018
Grant date	Jan 16, 2018

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributed data management. One of the methods includes receiving a plurality of feeds of streaming event data and routing feed data from each of the feeds to a respective channel of a plurality of channels, each of the channels being configured to store feed data until the feed data is consumed by a data sink, including routing feed data from a first feed to a first channel. A load metric for the first channel is determined to exceed a threshold. In response, a second channel is allocated for the first feed and feed data is redirected from the first feed to the second channel instead of the first channel.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, in a distributed computing system comprising a plurality of computers, a plurality of feeds of streaming event data of a multi-tenant application having a plurality of tenants, wherein each feed of the plurality of feeds is generated by a respective tenant of the plurality of tenants of the multi-tenant application;
routing feed data from each of the tenants of the multi-tenant application to a different respective channel of a plurality of channels, wherein each channel comprises a software process executing on one of the plurality of computers, wherein each channel of the plurality of channels is configured to receive and store feed data until the feed data is consumed by a data sink;
computing a load metric for a first channel executing on a first computer of the plurality of computers, wherein the first channel is allocated to receive first feed data generated by a first tenant of the plurality of tenants of the multi-tenant application;
determining that the load metric for the first channel exceeds a threshold;
in response to determining that the load metric for the first channel exceeds the threshold, selecting, from among different quantities of allocable channels, a particular quantity of new channels for the first feed based on an amount by which the load metric exceeds the threshold;
allocating the particular quantity of new channels for the first feed data generated by the first tenant of the plurality of tenants of the multi-tenant application; and
redirecting feed data from the first tenant to the new channels instead of the first channel.

2. The method of claim 1 , further comprising:
before the first feed data is received from the first tenant of the plurality of tenants of the multi-tenant application, receiving a message from the first tenant; and
in response to receiving the message from the first tenant:
allocating the first channel for the first tenant so that the first channel receives only first feed data from the first tenant;
allocating a networking connection for the first tenant, the networking connection being configured to receive first feed data from the first tenant; and
associating the networking connection with the first channel so that the distributed computing system routes the first feed data from the first tenant from the networking connection to the first channel.

3. The method of claim 2 , wherein redirecting the first feed data generated by the first tenant of the plurality of tenants of the multi-tenant application to the second channel includes associating the networking connection with the second channel.

4. The method of claim 3 , further comprising, after associating the networking connection with the second channel, deallocating the first channel so that one or more computing resources of the distributed computing system used by the first channel become available for one or more other channels.

5. The method of claim 1 , wherein computing the load metric for the first channel comprises determining an amount of memory or network bandwidth that is being used to execute the first channel on the first computer of the plurality of computers.

6. The method of claim 1 , wherein the data sink is a distributed file system for the distributed computing system, wherein the method further comprises sinking a respective output from each of the channels to the distributed file system for the distributed computing system.

7. The method of claim 1 , wherein the data sink is a data parsing engine configured to perform data analytics on feed data generated by tenants of the multi-tenant application, wherein the method further comprises sinking a respective output from each of the channels to the data parsing engine.

8. The method of claim 1 , wherein each of the plurality of channels is configured to receive and store feed data in storage that is local to one or more computers or one or more clusters of computers in the distributed computing system, wherein the storage comprises random access memory (RAM) or mass storage or both.

9. The method of claim 1 , wherein an instance of the multi-tenant application runs on the distributed computing system, wherein each tenant of the plurality of tenants is provided with access to the instance of the multi-tenant application that is running on the distributed computing system.

10. The method of claim 1 , wherein the multi-tenant application has a software architecture that includes a plurality of different layers, wherein each tenant of the plurality of tenants is provided with access to a different respective set of one or more layers of the multi-tenant application's software architecture.

11. The method of claim 1 , wherein the first channel comprises a first software process executing on the first computer of the plurality of computers, wherein the method further comprises:
after allocating the second channel, determining that the data sink has consumed all of the first feed data having been stored by the first channel; and
in response to determining that the data sink has consumed all of the first feed data having been stored by the first channel, terminating execution of the first software process on the first computer of the plurality of computers.

12. A distributed computing system comprising a plurality of physical computers configured to perform operations comprising:
receiving a plurality of feeds of streaming event data a plurality of feeds of streaming event data of a multi-tenant application having a plurality of tenants, wherein each feed of the plurality of feeds is generated by a respective tenant of the plurality of tenants of the multi-tenant application;
routing feed data from each of the tenants of the multi-tenant application to a different respective channel of a plurality of channels, wherein each channel comprises a software process executing on one of the plurality of computers, wherein each channel of the plurality of channels is configured to receive and store feed data until the feed data is consumed by a data sink;
computing a load metric for a first channel executing on a first computer of the plurality of computers, wherein the first channel is allocated to receive first feed data generated by a first tenant of the plurality of tenants of the multi-tenant application;
determining that the load metric for the first channel exceeds a threshold;
in response to determining that the load metric for the first channel exceeds the threshold, selecting, from among different quantities of allocable channels, a particular quantity of new channels for the first feed based on an amount by which the load metric exceeds the threshold;
allocating the particular quantity of new channels for the first feed data generated by the first tenant of the plurality of tenants of the multi-tenant application; and
redirecting feed data from the first tenant to the new channels instead of the first channel.

13. The distributed computing system of claim 12 , the operations further comprising:
before the first feed data is received from the first tenant of the plurality of tenants of the multi-tenant application, receiving a message from the first tenant; and
in response to receiving the message from the first tenant:
allocating the first channel for the first tenant so that the first channel receives only first feed data from the first tenant;
allocating a networking connection for the first tenant, the networking connection being configured to receive first feed data from the first tenant; and
associating the networking connection with the first channel so that the distributed computing sy

Assignees

• Pivotal Software Inc

Inventors

• Li Xiong
• Gao Xiaoming
• Li Ye
• Han Leng
• Zhu Lei

Classifications

• G06F2209/5022

  Workload threshold · CPC title

• G06F9/5083 Primary

  Techniques for rebalancing the load in a distributed system · CPC title

• G06F9/5011

  the resources being hardware resources other than CPUs, Servers and Terminals · CPC title