Inline wire speed deduplication system

Patent metadata

Field	Value
Publication number	US-10417233-B2
Application number	US-201615194331-A
Country	US
Kind code	B2
Filing date	Jun 27, 2016
Priority date	Jun 9, 2010
Publication date	Sep 17, 2019
Grant date	Sep 17, 2019

Abstract

Systems for performing inline wire speed data deduplication are described herein. Some embodiments include a device for inline data deduplication that includes one or more input ports for receiving an input data stream containing duplicates, one or more output ports for providing a data deduplicated output data stream, and an inline data deduplication engine coupled to said one or more input ports and said one or more output ports to process input data containing duplicates into output data which is data deduplicated, said inline data deduplication engine having an inline data deduplication bandwidth of at least 4 Gigabytes per second.

First claim

What is claimed is:

1. A device for inline data deduplication, comprising:
one or more input ports for receiving an input data stream containing duplicates;
one or more output ports for providing a data deduplicated output data stream; and
an inline data deduplication engine coupled to said one or more input ports and said one or more output ports to process input data containing duplicates into output data which is data deduplicated, said inline data deduplication engine having an inline data deduplication bandwidth of at least 4 Gigabytes per second.

2. The device of claim 1 , wherein said inline data deduplication engine:
extracts input data from the input data stream;
subdivides the extracted input data into input data chunks, the input data comprising duplicate input data chunks;
identifies input data chunks that do not match any previously processed data chunks already provided to the destination device; and
provides non-matching input data chunks to an output port of the one or more output ports as deduplicated data for inclusion in the output data stream.

3. The device of claim 1 , wherein said inline data deduplication engine comprises:
frame memory comprising at least some of the received input data stream and at least some output data provided for inclusion in the output data stream;
chunking logic for subdividing input data extracted from the input data stream into input data chunks; and
chunk identifier logic for generating a chunk identifier for each of the input data chunks based at least in part upon data within the input data chunk;
wherein each chunk identifier is uniquely associated with a particular sequence of chunk data.

4. The device of claim 3 , wherein the input data chunks are variable in size, and wherein the size of each input data chunk depends upon the content of the extracted input data.

5. The device of claim 3 , wherein said chunking logic processes the input data stream into input data chunks at a data rate of at least 400 Megabytes per second.

6. A data deduplication method performed by an inline deduplication engine, the method comprising:
receiving an input data stream containing duplicates;
providing a data deduplicated output data stream; and
processing input data containing duplicates into output data which is data deduplicated, said processing being performed at a rate of at least 4 Gigabytes per second.

7. The method of claim 6 , further comprising:
extracting input data from the input data stream;
subdividing the extracted input data into input data chunks, the input data comprising duplicate input data chunks;
identifying input data chunks that do not match any previously processed data chunks already provided to the destination device; and
providing non-matching input data chunks to an output port of the one or more output ports as deduplicated data for inclusion in the output data stream.

8. The method of claim 6 , further comprising:
subdividing input data extracted from an input data stream into input data chunks; and
generating a chunk identifier for each of the input data chunks based at least in part upon data within the input data chunk, each chunk identifier uniquely associated with a particular sequence of chunk data.

9. The method of claim 8 , wherein the input data chunks are variable in size, and wherein the size of each input data chunk depends upon the content of the extracted input data.

10. The method of claim 8 , wherein said subdividing the extracted input data is performed at a rate of at least 4 Megabytes per second.

11. A device for inline data deduplication, comprising:
one or more input ports for receiving an input data stream containing duplicates;
one or more output ports for providing a data deduplicated output data stream; and
an inline data deduplication engine coupled to said one or more input ports and said one or more output ports to process input data containing duplicates into output data which is data deduplicated, said inline data deduplication engine having an inline data deduplication bandwidth of at least 400 Megabytes per second per input port.

12. The device of claim 11 , wherein said inline data deduplication engine:
extracts input data from the input data stream;
subdivides the extracted input data into input data chunks, the input data comprising duplicate input data chunks;
identifies input data chunks that do not match any previously processed data chunks already provided to the destination device; and
provides non-matching input data chunks to an output port of the one or more output ports as deduplicated data for inclusion in the output data stream.

13. The device of claim 11 , wherein said inline data deduplication engine comprises:
frame memory comprising at least some of the received input data stream and at least some output data provided for inclusion in the output data stream;
chunking logic for subdividing input data extracted from the input data stream into input data chunks; and
chunk identifier logic for generating a chunk identifier for each of the input data chunks based at least in part upon data within the input data chunk;
wherein each chunk identifier is uniquely associated with a particular sequence of chunk data.

14. The device of claim 13 , wherein the input data chunks are variable in size, and wherein the size of each input data chunk depends upon the content of the extracted input data.

15. The device of claim 13 , wherein said chunking logic processes the input data stream into input data chunks at a data rate of at least 400 Megabytes per second.

16. A data deduplication method performed by an inline data deduplication engine, the method comprising:
receiving an input data stream containing duplicates;
providing a data deduplicated output data stream; and
processing input data containing duplicates into output data which is data deduplicated, said processing being performed at a rate of at least 400 Megabytes per second per input port of the inline data deduplication engine.

17. The method of claim 16 , further comprising:
extracting input data from the input data stream;
subdividing the extracted input data into input data chunks, the input data comprising duplicate input data chunks;
identifying input data chunks that do not match any previously processed data chunks already provided to the destination device; and
providing non-matching input data chunks to an output port of the one or more output ports as deduplicated data for inclusion in the output data stream.

18. The method of claim 16 , further comprising:
subdividing input data extracted from an input data stream into input data chunks; and
generating a chunk identifier for each of the input data chunks based at least in part upon data within the input data chunk, each chunk identifier uniquely associated with a particular sequence of chunk data.

19. The method of claim 18 , wherein the input data chunks are variable in size, and wherein the size of each input data chunk depends upon the content of the extracted input data.

20. The method of claim 18 , wherein said subdividing the extracted input data is performed at a rate of at least 4 Megabytes per second.

Assignees

• Avago Tech Int Sales Pte Lid

Inventors

• Sabaa Amr
• Kumar Pashupati
• Vu Bao
• Parekh Tarak
• Kuriakose Poulo
• Guntaka Vidyasagara Reddy
• Hans Madhsudan
• Ko Kung-Ling

Classifications

• G06F3/0641

  De-duplication techniques · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/24556 Primary

  Aggregation; Duplicate elimination · CPC title

• G06F16/1744

  using compression, e.g. sparse files · CPC title