Adaptive control of data collection requests sent to external data sources

Patent metadata

Field	Value
Publication number	US-10007710-B2
Application number	US-201615011525-A
Country	US
Kind code	B2
Filing date	Jan 30, 2016
Priority date	Sep 21, 2015
Publication date	Jun 26, 2018
Grant date	Jun 26, 2018

Abstract

Techniques and mechanisms are disclosed that enable a data collection system to adaptively control collection of data from one or more external data sources. At a high level, adaptively controlling collection of data from external data sources may include collecting performance information related to one or more data collection nodes and, in response to analyzing the collected performance information, adapting rates at which the data collection nodes send data collection requests to external data sources. Data collection performance information generally may include, but is not limited to, network traffic data, error messages generated by external data sources and/or data collection nodes, computing device performance information, and any other types of information related to a data collection node's ability to collect data from external data sources.

First claim

What is claimed is:

1. A method, comprising:
receiving, by a controller node of a data intake and query system, data collection performance information from a plurality of data collection nodes, the data collection performance information related to data collection requests sent by the plurality of data collection nodes to at least one component of a plurality of components of a cloud computing service, the plurality of components of the cloud computing service including distributed computing resources used to generate virtual machine instances, the data collection requests including a request for machine data generated by one or more components of the cloud computing service; generating, based on the data collection performance information, instructions for a data collection node of the plurality of data collection nodes to modify a rate at which the data collection node sends data collection requests to the at least one component of the cloud computing service; and sending the instructions to the data collection node.

2. The method of claim 1 , wherein the data collection requests are sent to an application programming interface (API) associated with the cloud computing service.

3. The method of claim 1 , wherein the instructions instruct the data collection node to increase the rate at which the data collection node sends the data collection requests to the at least one component of the cloud computing service.

4. The method of claim 1 , wherein the instructions instruct the data collection node to decrease the rate at which the data collection node sends the data collection requests to the at least one component of the cloud computing service.

5. The method of claim 1 , wherein the data collection performance information includes information indicating one or more error messages received by the data collection node from the at least one component of the cloud computing service.

6. The method of claim 1 , wherein the data collection performance information includes network traffic data.

7. The method of claim 1 , wherein the data collection performance information includes CPU utilization information for the data collection node.

8. The method of claim 1 , wherein the data collection performance information includes information indicating one or more attributes of the cloud computing service.

9. The method of claim 1 , wherein the data collection performance information includes information indicating one or more data collection policies of the cloud computing service.

10. The method of claim 1 , wherein the controller node receives particular data collection performance information from each data collection node of the plurality of data collection nodes.

11. The method of claim 1 , wherein the controller node receives particular data collection performance information from each data collection node of the plurality of data collection nodes, and wherein the plurality of data collection nodes send data collection requests to a plurality of cloud computing services.

12. The method of claim 1 , wherein the data collection node sends data collection requests to a plurality of cloud computing services.

13. The method of claim 1 , further comprising:
subsequent to generating the instructions, receiving additional data collection performance information from the data collection node;
generating, based at least in part on the additional data collection performance information, updated instructions for the data collection node to modify the rate at which the data collection node sends data collection requests to the at least one component of the cloud computing service; and
sending the updated instructions to the data collection node.

14. The method of claim 1 , wherein the data collection node sends data collection requests to at least one component of a plurality of cloud computing services, and wherein the controller node generates and sends to the data collection node separate instructions for each different cloud computing service of the plurality of cloud computing services.

15. The method of claim 1 , further comprising:
generating, based on the data collection performance information, second instructions for a different data collection node of the plurality of data collection nodes to send data collection requests to the at least one component of the cloud computing service; and sending the second instructions to the different data collection node.

16. The method of claim 1 , wherein the data collection requests sent to the at least one component of the cloud computing service request data related to performance and/or security information for one or more of the distributed computing resources of the cloud computing service.

17. The method of claim 1 , wherein data collected from the cloud computing service is sent to one or more indexers to be parsed and stored as event data reflecting events of one or more resources of an information technology environment.

18. The method of claim 1 , wherein data collected from the cloud computing service comprises raw data sent to one or more indexers, and wherein the one or more indexers parse a plurality of timestamped events from the raw data, and wherein each timestamped event of the plurality of timestamped events includes a portion of the raw data.

19. The method of claim 1 , further comprising:
receiving, by the data collection node, the instructions; and
sending, by the data collection node, data collection requests to the at least one component of the cloud computing service at a rate based on the instructions.

20. The method of claim 1 , further comprising:
receiving, by the controller node, an indication that new data collection node has joined the plurality of data collection nodes; and
modifying instructions for one or more data collection nodes of the plurality of data collection nodes based on the indication that the new data collection node has joined the plurality of data collection nodes.

21. The method of claim 1 , further comprising:
receiving, by the controller node, an indication that data collection node has left the plurality of data collection nodes; and
modifying instructions for one or more data collection nodes of the plurality of data collection nodes based on the indication that a data collection node has left the plurality of data collection nodes.

22. The method of claim 1 , further comprising:
determining that the data collection performance information indicates that the cloud computing service has upgraded its data collection capabilities; and
in response to determining that the cloud computing service has upgraded its data collection capabilities, instructing the data collection node to increase the rate at which the data collection node sends the data collection requests to the at least one component of the cloud computing service.

23. The method of claim 1 , wherein the data collection performance information indicates that the cloud computing service has upgraded its data collection capabilities based on one or more of: one or more status messages, one or more version identifiers, an operational pattern of the distributed computing resources of the cloud computing service, and data retrieved from a website separate from the cloud computing service,
the method further comprising instructing the data collection node to increase the rate at which the data collection node sends the data collection requests to the at least one component of the cloud computing service in respo

Assignees

• Splunk Inc

Inventors

• Chen Ken
• Tao Gang
• Ding Lai Qiang
• Hao Junqing
• Wang Ting
• Haddad Elias
• Bitincka Dritan

Classifications

• H04L43/0817

  by checking functioning · CPC title

• G06F17/30554 Primary

  Physics · mapped topic

• H04L43/024

  by adaptive sampling · CPC title

• G06F16/248 Primary

  Presentation of query results · CPC title