Systems and methods for distributive data storage

Patent metadata

Field	Value
Publication number	US-10776014-B2
Application number	US-201916278327-A
Country	US
Kind code	B2
Filing date	Feb 18, 2019
Priority date	Jan 4, 2017
Publication date	Sep 15, 2020
Grant date	Sep 15, 2020

Abstract

Some embodiments can include a system. In some embodiments, a system can comprise one or more processors and one or more non-transitory storage devices storing computing instructions configured to run on the one or more processors and perform acts. In many embodiments, the acts can comprise dividing a computer file into two or more portions; calculating a portion hash value for each portion of the two or more portions, wherein each portion hash value for each portion of the two or more portions can be different than other portion hash values for other portions of the two or more portions; creating two or more computer file portion replicas of each portion of the two or more portions, wherein each of the two or more computer file portion replicas for a corresponding portion of the two or more portions can comprise a same portion hash value as the corresponding portion of the two or more portions; sending the two or more computer file portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes; storing each of the two or more computer file portion replicas in the two or more storage nodes. Other embodiments are disclosed herein.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more non-transitory storage devices storing computing instructions configured to run on the one or more processors and perform:
dividing a computer file into two or more portions;
calculating a portion hash value for each respective portion of the two or more portions, wherein each respective portion hash value for each respective portion of the two or more portions is different than other respective portion hash values for other respective portions of the two or more portions;
creating two or more respective computer file portion replicas of each respective portion of the two or more portions, wherein each respective computer file portion replica of the two or more respective computer file portion replicas:
is for a corresponding portion of the two or more portions; and
comprises a same portion hash value as the corresponding portion of the two or more portions;

sending the two or more respective computer file portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes; and
storing each respective computer file portion replica of the two or more respective computer file portion replicas in the two or more storage nodes.

2. The system of claim 1 , wherein:
dividing the computer file into the two or more portions comprises determining that a computer file size of the computer file is greater than 2 megabytes.

3. The system of claim 2 , wherein:
a respective portion size of each respective portion of the two or more portions is less than 2 megabytes.

4. The system of claim 1 , wherein the one or more non-transitory storage devices storing the computing instructions are further configured to run on the one or more processors and perform:
receiving, from an electronic device of a user, a request to utilize the computer file;
sending a request to retrieve the computer file from the distributive data storage cluster;
retrieving the computer file from the distributive data storage cluster by retrieving a set of the two or more respective computer file portion replicas, wherein the set of the two or more respective computer file portion replicas, when combined, comprises the computer file; and
transmitting instructions allowing the electronic device of the user to utilize the computer file.

5. The system of claim 4 , wherein the one or more non-transitory storage devices storing the computing instructions are further configured to run on the one or more processors and perform:
calculating a respective retrieved portion hash value for each respective portion of the two or more portions associated with the set of the two or more respective computer file portion replicas, as retrieved; and
determining that the respective retrieved portion hash value for each respective portion of the two or more portions associated with the set of the two or more respective computer file portion replicas, as retrieved, matches the respective portion hash value for each respective portion of the two or more portions, thereby confirming that the two or more respective computer file portion replicas, as retrieved, have not been corrupted.

6. The system of claim 4 , wherein the one or more non-transitory storage devices storing the computing instructions are further configured to run on the one or more processors and perform:
merging the set of the two or more respective computer file portion replicas, as retrieved, into the computer file.

7. The system of claim 4 , wherein:
at least one respective computer file portion replica of the two or more respective computer file portion replicas, as retrieved, comprise a fastest responding computer file portion replica.

8. The system of claim 1 , wherein:
the distributive data storage cluster comprises three storage nodes;
the two or more respective computer file portion replicas comprise three respective computer file portion replicas; and
storing each respective computer file portion replica of the two or more respective computer file portion replicas comprises:
storing one respective computer file portion replica of the three respective computer file portion replicas in each of the three storage nodes.

9. The system of claim 1 , wherein:
the computer file comprises an image file.

10. The system of claim 1 , wherein creating the two or more respective computer file portion replicas of each respective portion of the two or more portions comprises:
replicating each respective portion of the two or more portions by a respective replication factor based, at least in part, on a number of the two or more storage nodes in the distributive data storage cluster.

11. A method comprising:
dividing a computer file into two or more portions;
calculating a portion hash value for each respective portion of the two or more portions, wherein each respective portion hash value for each respective portion of the two or more portions is different than other respective portion hash values for other respective portions of the two or more portions;
creating two or more respective computer file portion replicas of each respective portion of the two or more portions, wherein each respective computer file portion replica of the two or more respective computer file portion replicas:
is for a corresponding portion of the two or more portions; and
comprises a same portion hash value as the corresponding portion of the two or more portions;

sending the two or more respective computer file portion replicas to a distributive data storage cluster, the distributive data storage cluster comprising two or more storage nodes; and
storing each respective computer file portion replica of the two or more respective computer file portion replicas in the two or more storage nodes.

12. The method of claim 11 , wherein:
dividing the computer file into the two or more portions comprises determining that a computer file size of the computer file is greater than 2 megabytes.

13. The method of claim 12 , wherein:
a respective portion size of each respective portion of the two or more portions is less than 2 megabytes.

14. The method of claim 11 further comprising:
receiving, from an electronic device of a user, a request to utilize the computer file;
sending a request to retrieve the computer file from the distributive data storage cluster;
retrieving the computer file from the distributive data storage cluster by retrieving a set of the two or more respective computer file portion replicas, wherein the set of the two or more respective computer file portion replicas, when combined, comprises the computer file; and
transmitting instructions allowing the electronic device of the user to utilize the computer file.

15. The method of claim 14 further comprising:
calculating a respective retrieved portion hash value for each respective portion of the two or more portions associated with the set of the two or more respective computer file portion replicas, as retrieved; and
determining that the respective retrieved portion hash value for each respective portion of the two or more portions associated with the set of the two or more respective computer file portion replicas, as retrieved, matches the respective portion hash value for each respective portion of the two or more portions, thereby confirming that the two or more respective computer file portion replicas, as retrieved, have not been corrupted.

16. The method of claim 14 , further comprising:
merging the set of the two or more respective computer file portion

Assignees

• Walmart Apollo Llc

Inventors

• Sehgal Charandeep
• Bhat Vikas
• Krishnan Ganesh
• Kandaswamy Venkatesh

Classifications

• G06F16/51 Primary

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F3/0619

  in relation to data integrity, e.g. data losses, bit errors · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0659

  Command handling arrangements, e.g. command buffers, queues, command scheduling · CPC title

• G06F3/0613 Primary

  in relation to throughput · CPC title