Hierarchichal sharding of flows from sensors to collectors

Patent metadata

Field	Value
Publication number	US-2016359708-A1
Application number	US-201615171855-A
Country	US
Kind code	A1
Filing date	Jun 2, 2016
Priority date	Jun 5, 2015
Publication date	Dec 8, 2016
Grant date	—

Abstract

Systems, methods, and computer-readable media for hierarchichal sharding of flows from sensors to collectors. A first collector can receive a first portion of a network flow from a first capturing agent and determine that a second portion of the network flow was not received from the first capturing agent. The first collector can then send the first portion of the network flow to a second collector. A third collector can receive the second portion of the network flow from a second capturing agent and determine that the third collector did not receive the first portion of the network flow. The third collector can then send the second portion of the network flow to the second collector. The second collector can then aggregate the first portion and second portion of the network flow to yield the entire portion of the network flow.

First claim

What is claimed is:

1 . A method comprising:
receiving, by a first collector device, a first portion of a network flow from a first capturing agent, the first portion of the network flow being captured by the first capturing agent at a first host associated with the first capturing agent; determining, by the first collector device, that a second portion of the network flow was not received, to yield a first determination; based on the first determination, sending, by the first collector device, the first portion of the network flow to a second collector device; receiving, by a third collector device, the second portion of the network flow from a second capturing agent, the second portion of the network flow being captured by the second capturing agent at a second host associated with the second capturing agent; determining, by the third collector device, that the first portion of the network flow was not received, to yield a second determination; based on the second determination, sending, by the second collector device, the second portion of the network flow to the second collector device; determining, by the second collector device, that the first portion of the network flow and the second portion of the network flow are part of a same network flow; and aggregating, by the second collector device, the first portion of the network flow and the second portion of the network flow to yield an aggregated network flow.

2 . The method of claim 1 , further comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collector devices from the plurality of collector devices.

3 . The method of claim 2 , wherein the first portion of the network flow is transmitted by the first capturing agent to a first shard from the respective shards, and the second portion of the network flow is transmitted by the second capturing agent to a second shard from the respective shards, wherein the first collector device is part of the first shard and the third collector device is part of the second shard.

4 . The method of claim 2 , wherein the second collector device is part of a shard from the respective shards, and wherein the second collector device is assigned a flow key and hash that corresponds to the network flow.

5 . The method of claim 4 , wherein the second collector device is selected to receive the first portion and the second portion of the network flow from a plurality of other collector devices in the shard based on the flow key and hash corresponding to the network flow.

6 . The method of claim 1 , wherein the first collector device and the second collector device are mapped to respective shards from a first layer of shards, each of the respective shards comprising a selected group of collector devices, and wherein the second collector device is mapped to a shard from a second layer of shards.

7 . The method of claim 6 , wherein the first layer of shards is designated to receive network flow data directly from capturing agents deployed throughout a network, and wherein the second layer of shards is designated to receive different portions of the network flow data directly from different collector devices in the first layer of shards.

8 . The method of claim 1 , wherein the capturing agent comprises at least one of a process, a kernel module, or a software driver.

9 . The method of claim 8 , wherein the capturing agent resides in one of a hypervisor on the host or a virtual machine on the hypervisor, the network flow being sent or received by the one of the hypervisor or the virtual machine.

10 . The method of claim 1 , wherein the indication of network address spaces associated with the network comprises a mapping of network address spaces in the network, and wherein the network address spaces comprise at least one of:
one or more internet protocol addresses; one or more internet protocol address ranges; and one or more prefixes.

11 . A system comprising:
one or more processors; and one or more computer-readable storage devices having stored therein instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, by a first collector device, a first portion of a network flow from a first capturing agent, the first portion of the network flow being captured by the first capturing agent at a first host associated with the first capturing agent;
determining, by the first collector device, that a second portion of the network flow was not received, to yield a first determination;
based on the first determination, sending, by the first collector device, the first portion of the network flow to a second collector device;
receiving, by a third collector device, the second portion of the network flow from a second capturing agent, the second portion of the network flow being captured by the second capturing agent at a second host associated with the second capturing agent;
determining, by the third collector device, that the first portion of the network flow was not received, to yield a second determination;
based on the second determination, sending, by the second collector device, the second portion of the network flow to the second collector device;
determining, by the second collector device, that the first portion of the network flow and the second portion of the network flow are part of a same network flow; and
aggregating, by the second collector device, the first portion of the network flow and the second portion of the network flow to yield an aggregated network flow.

12 . The system of claim 11 , the one or more computer-readable storage devices storing additional instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collectors in the respective shards, wherein each of the respective shards comprises a number of assigned collector devices from the plurality of collector devices.

13 . The system of claim 12 , wherein the first portion of the network flow is transmitted by the first capturing agent to a first shard from the respective shards, and the second portion of the network flow is transmitted by the second capturing agent to a second shard from the respective shards, wherein the first collector device is part of the first shard and the third collector device is part of the second shard.

14 . The system of claim 12 , wherein the second collector device is part of a shard from the respective shards, and wherein the second collector device is assigned a flow key and hash that corresponds to the network flow.

15 . The system of claim 14 , wherein the second collector device is selected to receive the first portion and the second portion of the network flow from a plurality of other collector devices in the shard based on the flow key and hash corresponding to the network flow.

16 . A computer-readable storage device storing instructions which, when executed by a processor, cause the processor to perform operations comprising:
re

Assignees

• Cisco Tech Inc

Inventors

• Gandham Shashidhar
• Prasad Rohit Chandra
• Singh Abhishek Ranjan
• Yadav Navindra
• Deen Khawar
• Malhotra Varun Sagar

Classifications

• G06T11/26

  Drawing of charts or graphs · CPC title

• H04W72/54

  based on quality criteria · CPC title

• H04L41/0894

  Policy-based network configuration management · CPC title

• H04L43/0888

  Throughput · CPC title

• H04L9/0866

  involving user or device identifiers, e.g. serial number, physical or biometrical information, DNA, hand-signature or measurable physical characteristics · CPC title