Hierarchichal sharding of flows from sensors to collectors

Patent metadata

Field	Value
Publication number	US-11477097-B2
Application number	US-202016930116-A
Country	US
Kind code	B2
Filing date	Jul 15, 2020
Priority date	Jun 5, 2015
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Systems, methods, and computer-readable media for hierarchichal sharding of flows from sensors to collectors. A first collector can receive a first portion of a network flow from a first capturing agent and determine that a second portion of the network flow was not received from the first capturing agent. The first collector can then send the first portion of the network flow to a second collector. A third collector can receive the second portion of the network flow from a second capturing agent and determine that the third collector did not receive the first portion of the network flow. The third collector can then send the second portion of the network flow to the second collector. The second collector can then aggregate the first portion and second portion of the network flow to yield the entire portion of the network flow.

First claim

What is claimed is:

1. A method for recombining a network flow, the method comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors;
receiving, by a first collector, a first portion of the network flow;
receiving, at a second collector, a second portion of the network flow from a third collector that received the second portion of the network flow but not the first portion of the network flow;
determining, by the first collector, that the second portion of the network flow was not received at the first collector;
in response to determining the second portion was not received, sending, by the first collector, the first portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow;
wherein the second collector is part of a second shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow;
wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors based on the flow key and hash corresponding to the network flow.

2. The method of claim 1 , wherein the first portion of the network flow is transmitted by a first capturing agent to a first shard from the respective shards, wherein the first collector is part of the first shard.

3. The method of claim 1 , wherein the first collector is mapped to a respective shard from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.

4. A non-transitory computer-readable storage medium storing instructions to recombine a network flow, which when executed by at least one processor cause the at least one processor to perform operations comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors;
receiving, by a first collector, a first portion of the network flow;
receiving, at a second collector, a second portion of the network flow from a third collector that received the second portion of the network flow but not the first portion of the network flow;
determining, by the first collector, that the second portion of the network flow was not received at the first collector;
in response to determining the second portion was not received, sending, by the first collector, the first portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow;
wherein the second collector is part of a second shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow;
wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors based on the flow key and hash corresponding to the network flow.

5. The non-transitory computer-readable storage medium of claim 4 , wherein the first portion of the network flow is transmitted by a first capturing agent to a first shard from the respective shards, wherein the first collector is part of the first shard.

6. The non-transitory computer-readable storage medium of claim 4 , wherein the first collector is mapped to a respective shard from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.

7. A system that recombines a network flow, the system comprising:
at least one processor; and
at least one non-transitory computer-readable storage medium having stored therein instructions, which when executed by the at least one processor, causes the at least one processor to perform operations comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collectors;
receiving, by a first collector, a first portion of the network flow;
receiving, at a second collector, a second portion of the network flow from a third collector that received the second portion of the network flow but not the first portion of the network flow;
determining, by the first collector, that the second portion of the network flow was not received at the first collector;
in response to determining the second portion was not received, sending, by the first collector, the first portion of the network flow to the second collector; and
combining, by the second collector, the first portion of the network flow and the second portion of the network flow;
wherein the second collector is part of a second shard from the respective shards, and wherein the second collector is assigned a flow key and hash that corresponds to the network flow;
wherein the second collector is selected to receive the first portion and the second portion of the network flow from a plurality of other collectors based on the flow key and hash corresponding to the network flow.

8. The system of claim 7 , wherein the first portion of the network flow is transmitted by a first capturing agent to a first shard from the respective shards, wherein the first collector is part of the first shard.

9. The system of claim 7 , wherein the first collector is mapped to a respective shard from a first layer of shards, each of the respective shards comprising a selected group of collectors, and wherein the second collector is mapped to a shard from a second layer of shards.

Assignees

• Cisco Tech Inc

Inventors

• Gandham Shashidhar
• Prasad Rohit Chandra
• Singh Abhishek Ranjan
• Yadav Navindra
• Deen Khawar
• Malhotra Varun Sagar

Classifications

• G06T11/26

  Drawing of charts or graphs · CPC title

• H04W72/54

  based on quality criteria · CPC title

• H04L41/0894

  Policy-based network configuration management · CPC title

• G06F16/137

  Hash-based (content-based indexing of textual data G06F16/31) · CPC title

• G06F16/174

  Redundancy elimination performed by the file system (error detection or correction of the data by redundancy in operations G06F11/14) · CPC title