Dynamic configuration of remote capture agents for network data capture

Patent metadata

Field	Value
Publication number	US-9923767-B2
Application number	US-201414253744-A
Country	US
Kind code	B2
Filing date	Apr 15, 2014
Priority date	Apr 15, 2014
Publication date	Mar 20, 2018
Grant date	Mar 20, 2018

Abstract

The disclosed embodiments provide a method and system for facilitating the processing of network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network packets at the remote capture agent. Upon receiving an update to the configuration information from the configuration server, the system uses the update to reconfigure the generation of the event data by the remote capture agent during runtime of the remote capture agent.

First claim

What is claimed is:

1. A computer-implemented method performed by a remote capture agent coupled to a network, the method comprising:
obtaining configuration information from a configuration server over the network, the configuration information specifying a plurality of event streams to be generated by the remote capture agent and further specifying a respective event type associated with each event stream of the plurality of event streams;
monitoring network traffic comprising a plurality of network packets;
generating, based on the configuration information, a plurality of events from the network traffic, wherein generating an event of the plurality of events comprises:
extracting network packet data from at least one network packet of the plurality of network packets and associating the extracted network packet data with the event;
applying a filtering rule to the extracted network packet data to determine an event type associated with the event;
adding, based on the determined event type, the event to at least one event stream of the plurality of event streams;

for each event stream of the plurality of event streams:
selecting, based on the event type for the event stream specified in the configuration information, a component of a plurality of components on the network to which to send the event stream; and
sending the event stream to the selected component on the network for subsequent processing.

2. The computer-implemented method of claim 1 , wherein the selected component is a transformation server used to further transform the event data.

3. The computer-implemented method of claim 1 , further comprising:
transforming an event of the plurality of events into a transformed event; and
sending an event stream of the plurality of event streams containing the transformed event to one or more transformation servers.

4. The computer-implemented method of claim 1 , wherein the configuration information specifies one or more transformations of the event data, and wherein the one or more transformations comprise at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

5. The computer-implemented method of claim 1 , wherein the configuration information is obtained using at least one of a push mechanism and a pull mechanism.

6. The computer-implemented method of claim 1 , wherein the configuration information comprises at least one of an identifier for an event stream, a description for the event stream, an event stream type for the event stream, a custom field for the event stream, and an additional parameter for the event stream.

7. The computer-implemented method of claim 1 , wherein the event data comprises at least one of a transaction type, a timestamp, and an error indicator.

8. The computer-implemented method of claim 1 , wherein the configuration information comprises an additional parameter, and wherein the additional parameter is at least one of a time interval between events, a maximum number of aggregated events, and an inclusion of a matching transaction or matching error in the event data.

9. The computer-implemented method of claim 1 , wherein the remote capture agent is installed in a virtual computing environment.

10. The computer-implemented method of claim 1 , wherein at least one event stream of the plurality of event streams corresponds to event data including one or more of the following:
clickstream events;
HTTP transactions;
business transactions;
errors;
alerts; and
classified transactions.

11. The computer-implemented method of claim 1 , further comprising:
obtaining an update to the configuration information from the configuration server; and
generating, based on the updated configuration information, additional events from the network traffic.

12. A computer-implemented method performed by a configuration server coupled to a network, the method comprising:
obtaining configuration information for a set of remote capture agents on a set of networks, the configuration information specifying a plurality of event streams to be generated by each remote capture agent of the set of remote capture agents and further specifying a respective event type associated with each event stream of the plurality of event streams;
sending the configuration information to at least one remote capture agent of the set of remote capture agents, the configuration information causing the at least one remote capture agent of the set of remote capture agents to generate a plurality of events from network traffic, wherein generating an event of the plurality of events comprises:
extracting network packet data from at least one network packet and associating the extracted network packet data with the event;
applying a filtering rule to the extracted network packet data to determine an event type associated with the event; and
adding, based on the determined event type, the event to at least one event stream of the plurality of event streams;

wherein the configuration information further causes the at least one remote capture agent of the set of remote capture agents to, for each event stream of the plurality of event streams:
select, based on the event type for the event stream specified in the configuration information, a component of a plurality of components on the network to which to send the event stream; and
send the event stream to the selected component on the network for subsequent processing.

13. The computer-implemented method of claim 12 , further comprising:
obtaining an update to the configuration information at the configuration server; and
sending the updated configuration information to the remote capture agents, wherein the update is used by the remote capture agents to reconfigure the generation of the event data during runtime of the remote capture agents.

14. The computer-implemented method of claim 12 , wherein the configuration information is sent to the remote capture agents using at least one of a push mechanism and a pull mechanism.

15. The computer-implemented method of claim 12 , wherein the configuration information is obtained from an application used to access the event data after the event data is generated.

16. The computer-implemented method of claim 12 , wherein the method further comprises:
storing event data from the plurality of event streams in a data store; and
while subsequently processing a query,
employing a retrieval schema, which includes an extraction rule that indicates how to extract one or more values from an event, to extract values from event data stored in the data store; and
identifying responsive events based on the extracted values; and
wherein the retrieval schema is a late-binding retrieval schema that is applied during query execution.

17. The computer-implemented method of claim 12 , wherein at least one event stream of the plurality of event streams corresponds to event data including one or more of the following:
clickstream events;
HTTP transactions;
business transactions;
errors;
alerts; and
classified transactions.

18. The computer-implemented method of claim 12 , further comprising:
obtaining an update to the configuration information for the set of remote capture agents on the set of networks; and
sending the updated configuration information to at least one remote capture agent of the set of remote capture agents, the updated configuration information causing the at least one remote capture agent of the set of remote capture agents to generate additional events from the network traffic.

Assignees

• Splunk Inc

Inventors

• Dickey Michael

Classifications

• H04L41/0816 Primary

  the condition being an adaptation, e.g. in response to network events · CPC title

• H04L41/0856 Primary

  by backing up or archiving configuration information · CPC title