Hyper-scale p2p deduplicated storage system using a distributed ledger

Patent metadata

Field	Value
Publication number	US-2021019232-A1
Application number	US-201916516109-A
Country	US
Kind code	A1
Filing date	Jul 18, 2019
Priority date	Jul 18, 2019
Publication date	Jan 21, 2021
Grant date	—

Abstract

One example method includes receiving from a node, in an HSAN that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node, performing a challenge-and-response process with the node to verify that the node has a copy of the data that was the subject of the entry, making a determination that a replication factor X has not been met, and adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.

First claim

What is claimed is:

1 . A method, comprising:
receiving from a node, in an HSAN that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node; performing a challenge-and-response process with the node to verify that the node has a copy of the data that was the subject of the entry; making a determination that a replication factor X has not been met; and adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.

2 . The method as recited in claim 1 , further comprising splitting backup data into a plurality of data segments that includes the data segment, and hashing the data segment.

3 . The method as recited in claim 1 , wherein the distributed ledger does not permit an HSAN node access to the data segment upon which the entry is based unless that HSAN node has demonstrated possession of a copy of that data segment.

4 . The method as recited in claim 1 , wherein the distributed ledger is a public distributed ledger that permits any HSAN node to access the entry.

5 . The method as recited in claim 1 , wherein the distributed ledger is a public distributed ledger that includes one or more entries that point to public data accessible by all nodes of the HSAN.

6 . The method as recited in claim 1 , further comprising adding a node to the HSAN upon mutual agreement of other nodes of the HSAN.

7 . The method as recited in claim 1 , wherein metadata concerning the data segment identified in the request is omitted from the request.

8 . The method as recited in claim 1 , wherein use of the replication factor X ensures that no more than X copies of a data segment will be stored in the HSAN.

9 . The method as recited in claim 1 , wherein entries are stored in the distributed ledger as a blockchain.

10 . The method as recited in claim 1 , further comprising performing any one or more of the HSAN operations: IS_PRESENT; REF_DATA; GET_DATA; DEREF_DATA; DEL_DATA; CHECK_DATA; and/or REPL_DATA.

11 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving from a node, in an HSAN that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node; performing a challenge-and-response process with the node to verify that the node has a copy of the data that was the subject of the entry; making a determination that a replication factor X has not been met; and adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.

12 . The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise splitting backup data into a plurality of data segments that includes the data segment, and hashing the data segment.

13 . The non-transitory storage medium as recited in claim 11 , wherein the distributed ledger does not permit an HSAN node access to the data segment upon which the entry is based unless that HSAN node has demonstrated possession of a copy of that data segment.

14 . The non-transitory storage medium as recited in claim 11 , wherein the distributed ledger is a public distributed ledger that permits any HSAN node to access the entry.

15 . The non-transitory storage medium as recited in claim 11 , wherein the distributed ledger is a public distributed ledger that includes one or more entries that point to public data accessible by all nodes of the HSAN.

16 . The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise adding a node to the HSAN upon mutual agreement of other nodes of the HSAN.

17 . The non-transitory storage medium as recited in claim 11 , wherein metadata concerning the data segment identified in the request is omitted from the request.

18 . The non-transitory storage medium as recited in claim 11 , wherein use of the replication factor X ensures that no more than X copies of a data segment will be stored in the HSAN.

19 . The non-transitory storage medium as recited in claim 11 , wherein entries are stored in the distributed ledger as a blockchain.

20 . The non-transitory storage medium as recited in claim 11 , further comprising performing any one or more of the HSAN operations: IS_PRESENT; REF_DATA; GET_DATA; DEREF_DATA; DEL_DATA; CHECK_DATA; and/or REPL_DATA.

Assignees

• Emc Ip Holding Co Llc

Inventors

• Murti Arun
• Lei Joey C
• Brenner Adam E
• Malamut Mark D

Classifications

• H04L9/50

  using hash chains, e.g. blockchains or hash trees · CPC title

• G06F16/1834

  implemented based on peer-to-peer networks, e.g. gnutella (p2p communication protocols H04L67/104) · CPC title

• G06F16/1748 Primary

  De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641) · CPC title

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• G06F11/1453 Primary

  using de-duplication of the data · CPC title