Transformation of network data at remote capture agents

Patent metadata

Field	Value
Publication number	US-9762443-B2
Application number	US-201414253753-A
Country	US
Kind code	B2
Filing date	Apr 15, 2014
Priority date	Apr 15, 2014
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

The disclosed embodiments provide a method and system for processing network data. During operation, the system obtains, at a remote capture agent, configuration information for the remote capture agent from a configuration server over a network. Next, the system uses the configuration information to configure the generation of event data from network data obtained from network packets at the remote capture agent. The system then uses the configuration information to configure transformation of the event data or the network data into transformed event data at the remote capture agent.

First claim

What is claimed is:

1. A computer-implemented method performed by a remote capture agent coupled to a network, comprising:
obtaining configuration information from a configuration server over a network, wherein the configuration information is usable by the remote capture agent to generate timestamped event data from network packets and to transform the timestamped event data into transformed event data;
monitoring network traffic comprising a plurality of network packets;
generating, based on the configuration information, timestamped event data from at least one network packet of the plurality of network packets, wherein generating the timestamped event data includes segmenting the at least one network packet into a plurality of events and associating each event of the plurality of events with a respective timestamp; and
transforming, based on the same configuration information, the timestamped event data into transformed event data, wherein transforming the timestamped event data includes performing an operation involving data contained in at least one event of the plurality of events.

2. The computer-implemented method of claim 1 , further comprising:
sending an event stream comprising the transformed event data to one or more stream servers for further transformation of the transformed event data by the one or more stream servers.

3. The computer-implemented method of claim 2 , wherein the event stream includes a type of event data specified in the configuration information.

4. The computer-implemented method of claim 1 , further comprising:
receiving an update to the configuration information from the configuration server; and
generating, based on the updated configuration information, timestamped event data from at least one network packet of the plurality of network packets.

5. The computer-implemented method of claim 1 , wherein transformation of the timestamped event data comprises at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

6. The computer-implemented method of claim 1 , wherein transformation of the timestamped event data comprises:
obtaining a time interval associated with the timestamped event data or the network data; and
aggregating the timestamped event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

7. The computer-implemented method of claim 1 , wherein the configuration server obtains the configuration information from an application used to access the transformed event data.

8. The computer-implemented method of claim 1 , wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

9. The computer-implemented method of claim 1 , wherein the remote capture agent is installed in a virtual computing environment.

10. The computer-implemented method of claim 1 , wherein the transformation of the timestamped event data comprises:
identifying a network address in the timestamped event data;
identifying related data in a lookup table based on the identified network address; and
including the related data in the transformed event data.

11. A remote capture agent, comprising:
a processor;
a memory storing instructions which, when executed by the processor, cause the remote
capture agent to:
obtain configuration information from a configuration server over a network,
wherein the configuration information is usable by the remote capture agent to generate timestamped event data from network packets and to transform the timestamped event data into transformed event data;

monitor network traffic comprising a plurality of network packets;
generate, based on the configuration information, timestamped event data based on data contained in at least one network packet of the plurality of network packets; and
transform, based on the same configuration information, the timestamped event data into transformed event data.

12. The remote capture agent of claim 11 , wherein the instructions, when executed by the processor, further cause the remote capture agent to:
send an event stream comprising the transformed event data to one or more transformation servers for further transformation of the transformed event data by the one or more transformation servers.

13. The remote capture agent of claim 12 , wherein the event stream includes a type of event data specified in the configuration information.

14. The remote capture agent of claim 11 , wherein transformation of the timestamped event data comprises at least one of an aggregation, a calculation, a filter, a normalization, and a formatting.

15. The remote capture agent of claim 11 , wherein transformation of the timestamped event data comprises:
obtaining a time interval associated with the timestamped event data or the network data; and
aggregating the timestamped event data or the network data within the time interval into at least one of an event count, a statistic, and a uniqueness count.

16. The remote capture agent of claim 11 , wherein the configuration server obtains the configuration information from an application used to access the transformed event data.

17. The remote capture agent of claim 11 , wherein the configuration information comprises at least one of an identifier, a description, an event stream type, a custom field, and an additional parameter.

18. The remote capture agent of claim 11 , wherein the transformation of the event data or the network data comprises:
identifying a network address in the timestamped event data;
identifying related data in a lookup table based on the identified network address; and
including the related data in the transformed event data.

19. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause a remote capture agent coupled to a network to perform a method comprising:
obtaining configuration information from a configuration server over a network, wherein the configuration information is usable by the remote capture agent to generate timestamped event data from network packets and to transform the timestamped event data into transformed event data;
monitoring network traffic comprising a plurality of network packets;
generating, based on the configuration information, timestamped event data from at least one network packet of the plurality of network packets, wherein generating the timestamped event data includes segmenting the at least one network packet into a plurality of events and associating each event of the plurality of events with a respective timestamp; and
transforming, based on the same configuration information, the timestamped event data into transformed event data, wherein transforming the timestamped event data includes performing an operation involving data contained in at least one event of the plurality of events.

20. The non-transitory computer-readable storage medium of claim 19 , the method further comprising:
sending an event stream comprising the transformed event data to one or more stream servers for further transformation of the transformed event data by the one or more stream servers.

21. The non-transitory computer-readable storage medium of claim 20 , wherein the event stream includes a type of event data specified in the configuration information.

22. The non-transitory computer-readable storage medium of claim 19 , wherein transfo

Assignees

• Splunk Inc

Inventors

• Dickey Michael

Classifications

• H04L41/0816

  the condition being an adaptation, e.g. in response to network events · CPC title

• H04L43/04 Primary

  Processing captured monitoring data, e.g. for logfile generation · CPC title

• H04L41/046

  comprising network management agents or mobile agents therefor · CPC title

• H04L43/106

  using time related information in packets, e.g. by adding timestamps · CPC title

• H04L41/0856 Primary

  by backing up or archiving configuration information · CPC title