Flow-level deduplication of network traffic in a network traffic visibility system

Patent metadata

Field	Value
Publication number	US-12537683-B2
Application number	US-202418441400-A
Country	US
Kind code	B2
Filing date	Feb 14, 2024
Priority date	Feb 14, 2024
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

A system and method for flow-level deduplication of network traffic are disclosed. A network node receives a first plurality of packets from a first network endpoint. The first plurality of packets represent a flow of data being communicated between the first network endpoint and a second network endpoint. The network node further receives a second plurality of packets from the second network endpoint. The network node identifies a sequence identifier of each packet of the first and second pluralities of packets. The network node determines that the first and second pluralities of packets are all associated with the same flow, based on the sequence identifiers of the first and second pluralities of packets. In response to that determination, the network node deduplicates the flow by discarding the first plurality of packets or the second plurality of packets. The network node may be a traffic visibility node.

First claim

What is claimed is:

1 . A method comprising:
receiving, by a traffic visibility node, a first plurality of packets from a first network endpoint, wherein the first plurality of packets represents a flow of data being communicated between the first network endpoint and a second network endpoint; receiving, by the traffic visibility node, a second plurality of packets from the second network endpoint; identifying, by the traffic visibility node, a sequence identifier of each packet of the first plurality of packets and of each packet of the second plurality of packets; determining, by the traffic visibility node, that the first plurality of packets and the second plurality of packets are all associated with the same flow, based on the sequence identifiers of the first plurality of packets and the second plurality of packets, wherein the determining that the first plurality of packets and the second plurality of packets are all associated with the same flow comprises:
determining that the sequence identifiers of all of the first plurality of packets and the second plurality of packets are identical; and
reconstructing at least a portion of the flow at the traffic visibility node, by comparing at least a portion of data in the first plurality of packets with at least a portion of data in the second plurality of packets, within a sliding window; and in response to determining that the first plurality of packets and the second plurality of packets are all associated with the same flow, deduplicating the flow, by the traffic visibility node, by discarding at least a portion of the first plurality of packets or at least a portion of the second plurality of packets.

2 . The method of claim 1 , wherein determining that the first plurality of packets and the second plurality of packets are all associated with the same flow comprises determining that the sequence identifiers of all of the first plurality of packets and the second plurality of packets are identical.

3 . The method of claim 1 , wherein for each packet of the first plurality of packets and the second plurality of packets, the sequence identifier of the packet comprises a hash of a five-tuple and a directional indicator, the directional indicator being indicative of a communication direction of the packet.

4 . The method of claim 1 , wherein for each packet of the first plurality of packets and the second plurality of packets, the sequence identifier of the packet comprises a hash of header information from the packet, including source IP address, destination IP address, source port, destination port, protocol and a directional indicator, the directional indicator being indicative of a communication direction of the packet.

5 . The method of claim 1 , wherein the first plurality of packets is at least a portion of an SSL Read stream or an SSL Write stream synthesized at the first network endpoint.

6 . The method of claim 1 , wherein the first plurality of packets and the second plurality of packets are each at least a portion of an SSL Read stream or an SSL Write stream synthesized at the first network endpoint or the second network endpoint.

7 . The method of claim 1 , wherein:
the first plurality of packets and the second plurality of packets correspond to a flow of data being transmitted from the first network endpoint to the second network endpoint; the first plurality of packets is at least a portion of a synthesized SSL Write stream from the first network endpoint, corresponding to the flow of data being transmitted from the first network endpoint to the second network endpoint; and the second plurality of packets is at least a portion of a synthesized SSL Read stream from the second network endpoint, corresponding to the flow of data being transmitted from the first network endpoint to the second network endpoint.

8 . The method of claim 1 , wherein:
the first plurality of packets and the second plurality of packets correspond to a flow of data being transmitted from the second network endpoint to the first network endpoint; the first plurality of packets is at least a portion of a synthesized SSL Read stream from the first network endpoint, corresponding to the flow of data being transmitted from the second network endpoint and to the first network endpoint; and the second plurality of packets is at least a portion of a synthesized SSL Write stream from the second network endpoint, corresponding to the flow of data being transmitted from the first network endpoint and to second network endpoint.

9 . The method of claim 1 , wherein the deduplicating the flow results in a deduplicated flow, the method further comprising:
forwarding, by the traffic visibility node, at least a payload of a packet of the deduplicated flow to an external tool coupled to the traffic visibility node, for analysis.

10 . The method of claim 1 , wherein:
for each packet of the first plurality of packets and the second plurality of packets, the sequence identifier of the packet comprises a hash of header information from the packet, including source IP address, destination IP address, source port, destination port, protocol and a directional indicator, the directional indicator being indicative of a communication direction of the packet; the first plurality of packets and the second plurality of packets are each at least a portion of an SSL Read stream or an SSL Write stream synthesized at the first network endpoint or the second network endpoint; determining that the first plurality of packets and the second plurality of packets are all associated with the same flow comprises determining that the sequence identifiers of all of the first plurality of packets and the second plurality of packets are identical; and deduplicating the flow results in a deduplicated flow; the method further comprising: forwarding, by the traffic visibility node, at least a payload of a packet of the deduplicated flow to an external tool coupled to the traffic visibility node, for analysis.

11 . At least one machine-readable storage medium having instructions stored thereon, execution of which by at least one processor causes performance of operations comprising:
receiving, by a network node, a first plurality of packets from a first network endpoint that is external to the network node, wherein the first plurality of packets represents a flow of data being communicated between the first network endpoint and a second network endpoint that is external to the network node; receiving, by the network node, a second plurality of packets from the second network endpoint; identifying, by the network node, a sequence identifier of each packet of the first plurality of packets and of each packet of the second plurality of packets; determining, by the network node, that the first plurality of packets and the second plurality of packets are all associated with the same flow, based on the sequence identifiers of the first plurality of packets and the second plurality of packets, wherein the determining that the first plurality of packets and the second plurality of packets are all associated with the same flow comprises: determining that the sequence identifiers of all of the first plurality of packets and the second plurality of packets are identical; and reconstructing at least a portion of the flow at the network node, by comparing at least a portion of data in the first plurality of packets with at least a portion of data in the second plurality of packets, within a sliding window; and in response to determining that the first plurality of packets and the second plurality of packets are all associated with the same flow, deduplicating the flow, by the

Assignees

• Gigamon Inc

Inventors

• Bommana Murali
• Dahiya Sandeep
• Kumar Santhosh

Classifications

• H04L63/0428

  wherein the data content is protected, e.g. by encrypting or encapsulating the payload · CPC title

• H04L47/34

  ensuring sequence integrity, e.g. using sequence numbers · CPC title

• H04L43/0876

  Network utilisation, e.g. volume of load or congestion level · CPC title

• H04L9/32 Primary

  including means for verifying the identity or authority of a user of the system {or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials} · CPC title

• H04L41/40

  using virtualisation of network functions or resources, e.g. SDN or NFV entities · CPC title