Transforming event data using values obtained by querying a data source

Patent metadata

Field	Value
Publication number	US-11314737-B2
Application number	US-201816134778-A
Country	US
Kind code	B2
Filing date	Sep 18, 2018
Priority date	Apr 15, 2014
Publication date	Apr 26, 2022
Grant date	Apr 26, 2022

Abstract

The disclosed embodiments provide a method and system for processing network data. During operation, the system obtains one or more event streams from one or more remote capture agents over one or more networks, wherein the one or more event streams include event data generated from network packets captured by the one or more remote capture agents. Next, the system applies one or more transformations to the one or more event streams to obtain transformed event data from the event data. The system then enables querying of the transformed event data.

First claim

What is claimed is:

1. A computer-implemented method performed by a remote capture agent coupled to a network, the method comprising:
monitoring network traffic comprising a plurality of network packets;
generating timestamped event data based on the plurality of network packets;
querying a data source using at least one first value contained in a network packet of the plurality of network packets to obtain a related second value;
transforming, by the remote capture agent, the timestamped event data at least in part by including the related second value in the timestamped event data; and
sending, via a network, the timestamped event data including the related second value to another device on the network.

2. The computer-implemented method of claim 1 , wherein the data source includes data related to one or more client devices coupled to the network.

3. The computer-implemented method of claim 1 , wherein the first value contained in the network packet is an Internet Protocol (IP) address associated with a client device coupled to the network.

4. The computer-implemented method of claim 1 , the method further comprising storing, in a data store, the timestamped event data including the related second value.

5. The computer-implemented method of claim 1 , wherein the related second value includes one or more of: a name of a client device, a user identifier associated with the client device.

6. The computer-implemented method of claim 1 , wherein the data source is a first data source, and wherein the method further comprises:
using the related second value to query a second data source to obtain a related third value; and
transforming the timestamped event data at least in part by including the related third value in the timestamped event data.

7. The computer-implemented method of claim 1 , the method further comprising enabling querying of the timestamped event data, wherein enabling querying of the timestamped event data comprises:
indexing the timestamped event data; and
executing queries on the timestamped event data.

8. The computer-implemented method of claim 1 , the method further comprising:
indexing the timestamped event data; and
executing queries on the timestamped event data, wherein execution of a query is performed on different subsets of the timestamped event data by one or more indexers in parallel.

9. The computer-implemented method of claim 1 , wherein transforming the timestamped event data further includes at least one of: an aggregation, a calculation, a filter, a normalization, and a formatting.

10. The computer-implemented method of claim 1 , the method further comprising transmitting the timestamped event data over the network to a set of indexers, wherein the set of indexers are used to process queries using a late-binding schema of the timestamped event data.

11. A system used to improve processing of network data collected by a plurality of remote capture agents distributed across a network, comprising:
a remote capture agent implemented by a first computing device, the remote capture agent including instructions that upon execution cause the remote capture agent to:
monitor network traffic comprising a plurality of network packets;
generate timestamped event data based on the plurality of network packets;
send the timestamped event data to a transformation server; and

the transformation server implemented by a second computing device, the transformation server including instructions that upon execution cause the transformation server to:
query a data source using at least one first value contained in a network packet of the plurality of network packets to obtain a related second value;
transform the timestamped event data at least in part by including the related second value in the timestamped event data; and
send, via a network, the timestamped event data including the related second value to another device on the network.

12. The system of claim 11 , wherein the data source includes data related to one or more client devices coupled to the network.

13. The system of claim 11 , wherein the first value contained in the network packet is an Internet Protocol (IP) address associated with a client device coupled to the network.

14. The system of claim 11 , wherein the system further includes an indexer implemented by a third computing device, the indexer including instructions, upon execution, cause the indexer to store in a data store at least one of: the timestamped event data and the timestamped event data.

15. The system of claim 11 , wherein the related second value includes one or more of: a name of a client device, a user identifier associated with the client device.

16. The system of claim 11 , wherein the data source is a first data source, and wherein the transformation server includes further instructions that upon execution further cause the transformation server to:
use the related second value to query a second data source to obtain a related third value; and
transform the timestamped event data at least in part by including the related third value in the timestamped event data.

17. The system of claim 11 , further comprising an indexer implemented by a third computing device, the indexer including instructions that upon execution cause the indexer to enable querying of the timestamped event data, wherein enabling querying of the timestamped event data includes: indexing the timestamped event data, and executing queries on the timestamped event data.

18. The system of claim 11 , further comprising a plurality of indexers implemented by one or more third computing devices, each of the plurality of indexers including instructions that upon execution cause the indexer to:
index the timestamped event data; and
execute queries on the timestamped event data, wherein execution of a query is performed on different subsets of the timestamped event data by one or more indexers of the plurality of indexers in parallel.

19. The system of claim 11 , wherein transforming the timestamped event data further includes at least one of: an aggregation, a calculation, a filter, a normalization, and a formatting.

20. The system of claim 11 , further comprising an indexer implemented by a third computing device, the indexer including instructions that upon execution cause the indexer to process queries using a late-binding schema of the timestamped event data.

21. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause performance of operations comprising:
monitoring network traffic comprising a plurality of network packets;
generating timestamped event data based on the plurality of network packets;
querying a data source using at least one first value contained in a network packet of the plurality of network packets to obtain a related second value;
transforming, by a remote capture agent, the timestamped event data at least in part by including the related second value in the timestamped event data; and
sending, via a network, the timestamped event data including the related second value to another device on the network.

22. The non-transitory computer-readable storage medium of claim 21 , wherein the data source includes data related to one or more client devices coupled to the network.

23. The non-transitory computer-readable storage medium of claim 21 , wherein the first value contained in the network packet is an Internet Protoc

Assignees

• Splunk Inc

Inventors

• Dickey Michael

Classifications

• G06F16/245 Primary

  Query processing · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title