Facilitating custom content extraction from network packets

Patent metadata

Field	Value
Publication number	US-10334085-B2
Application number	US-201514609292-A
Country	US
Kind code	B2
Filing date	Jan 29, 2015
Priority date	Jan 29, 2015
Publication date	Jun 25, 2019
Grant date	Jun 25, 2019

Abstract

The disclosed embodiments provide a system for extracting custom content from network packets. During operation, the system receives a stream of packets. The system then parses packets in the stream to determine a protocol for each packet. Next, the system applies a custom-content-extraction rule to each packet associated with a target protocol to obtain the extracted content. Then, the system stores the extracted content in events in a data store to facilitate subsequent queries involving the extracted content.

First claim

What is claimed is:

1. A method performed by a remote capture agent coupled to a computer network, the method comprising:
monitoring a stream of network packets;
for each network packet of a plurality of network packets in the stream of network packets:
parsing the network packet to identify a structure of the network packet, the structure of the network packet used to determine a protocol associated with the network packet;
applying an extraction rule associated with the protocol to the network packet to obtain extracted content, wherein applying the extraction rule includes:
identifying at least one user-specified field in the network packet containing structured data from which the extracted content is to be obtained, and
extracting data from the structured data contained in the user-specified field of the network packet;

generating a timestamped event including a field storing the extracted content; and
sending the timestamped event including the extracted content to another component on the computer network for storage in a data store, the data store facilitating the querying of timestamped event data stored in the data store using late-binding schemas generated from received queries.

2. The method of claim 1 , wherein the method further comprises:
receiving a query to be applied to timestamped events stored in the data store;
retrieving timestamped events from the data store satisfying the query;
using a late-binding schema generated from the query to retrieve data values from the retrieved timestamped events; and
processing the query using the retrieved data values.

3. The method of claim 1 , wherein a rule for determining the protocol is different than the extraction rule that is used to obtain the extracted content from the network packet.

4. The method of claim 1 , wherein applying the extraction rule to the network packet includes applying a field-specific regular expression to the at least one user-specified field in the network packet.

5. The method of claim 1 , wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for an XML data format to the at least one user-specified field in the network packet.

6. The method of claim 1 , wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for a JSON data format to the at least one user-specified field in the network packet.

7. The method of claim 1 , wherein parsing the network packet to determine a protocol associated with the network packet includes using a deep-packet inspection engine to determine the protocol.

8. The method of claim 1 , wherein the extraction rule is specified by a user using a graphical user interface (GUI).

9. The method of claim 1 , wherein the extraction rule is specified by a user using a graphical user interface (GUI); and
wherein the GUI includes fields used to receive input from a user, wherein the input includes one or more of:
a source field identifier that identifies a source field in network packets of the stream of network packets from which to obtain the extracted content,
an extraction rule type that specifies a type of extraction rule to be used to obtain the extracted content,
an extraction rule definition, and
an identifier that is used to identify the extracted content.

10. The method of claim 1 , wherein the remote capture agent receives the extraction rule from a configuration server; and
wherein sending the timestamped event includes streaming the timestamped event from the remote capture agent to the data store.

11. A non-transitory computer-readable storage medium storing instructions that when executed by a processor cause performance of operations comprising:
monitoring a stream of network packets;
for each network packet of a plurality of network packets in the stream of network packets:
parsing the network packet to identify a structure of the network packet, the structure of the network packed used to determine a protocol associated with the network packet;
applying an extraction rule associated with the protocol to the network packet to obtain extracted content, wherein applying the extraction rule includes:
identifying at least one user-specified field in the network packet containing structured data from which the extracted content is to be obtained,
extracting data from the structured data contained in the user-specified field of the network packet;

generating a timestamped event including a field storing the extracted content; and
sending the timestamped event including the extracted content to another component on a computer network for storage in a data store, the data store facilitating the querying of timestamped event data stored in the data store using late-binding schemas generated from received queries.

12. The non-transitory computer-readable storage medium of claim 11 , further comprising instructions which, when executed by the processor, cause performance of operations comprising:
receiving a query to be applied to timestamped events stored in the data store;
retrieving timestamped events from the data store satisfying the query;
using a late-binding schema generated from the query to retrieve data values from the retrieved timestamped events; and
processing the query using the retrieved data values.

13. The non-transitory computer-readable storage medium of claim 11 , wherein a rule for determining the protocol is different than the extraction rule that is used to obtain the extracted content from the network packet that matches the determined protocol.

14. The non-transitory computer-readable storage medium of claim 11 , wherein applying the extraction rule to the network packet includes applying a field-specific regular expression to the at least one user-specified field in the network packet.

15. The non-transitory computer-readable storage medium of claim 11 , wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for an XML data format to the at least one user-specified field in the network packet.

16. The non-transitory computer-readable storage medium of claim 11 , wherein applying the extraction rule to the network packet includes applying a field-specific extraction rule for a JSON data format to the at least one user-specified field in the network packet.

17. The non-transitory computer-readable storage medium of claim 11 , wherein parsing the network packet to determine a protocol associated with the network packet includes using a deep-packet inspection engine to determine the protocol.

18. The non-transitory computer-readable storage medium of claim 11 , wherein the extraction rule is specified by a user using a graphical user interface (GUI).

19. The non-transitory computer-readable storage medium of claim 11 , wherein the extraction rule is specified by a user using a graphical user interface (GUI); and
wherein the GUI includes fields used to receive input from a user, wherein the input includes one or more of:
a source field identifier that identifies a source field in network packets of the stream of network packets from which to obtain the extracted content,
an extraction rule type that specifies a type of extraction rule to be used to obtain the extracted content,
an extraction rule definition, and
an identifier that is used to identify the extracted content.

20. The non-transitory computer-readable storage medium of claim 11 ,
wherein a remote capture agent

Assignees

• Splunk Inc

Inventors

• Hsiao Fang I
• Ching Clayton S
• Dickey Michael R
• Shcherbakov Vladimir A
• Sharp Clint

Classifications

• H04L43/028

  by filtering · CPC title

• H04L69/22 Primary

  Parsing or analysis of headers · CPC title

• H04L43/0876

  Network utilisation, e.g. volume of load or congestion level · CPC title