Object sharding in a host-side processing device for distributed storage

Patent metadata

Field	Value
Publication number	US-9898474-B1
Application number	US-201514589919-A
Country	US
Kind code	B1
Filing date	Jan 5, 2015
Priority date	Jan 5, 2015
Publication date	Feb 20, 2018
Grant date	Feb 20, 2018

Abstract

A host-side network processing device coupled between a storage system and a host machine may encode a data object according to a sharding technique. Encoding the data object may include receiving, from the host machine, a data object and a request to store the data object. The network processing device may send a request for a plurality of storage locations to the storage system. The network processing device may receive a location list including a plurality of location identifiers from the storage system, where each location identifier corresponds to a respective storage location. The network processing device may determine an encoding matrix and encode the data object into a plurality of shards based on the encoding matrix according to a sharding technique. The network processing device may generate a plurality of storage requests that each specify a different location identifier and that each includes a respective shard.

First claim

What is claimed is:

1. A system, comprising:
a storage system connected to a network, the storage system comprising a plurality of servers and a plurality of non-transitory storage devices, wherein the storage system is configured to store a plurality of data objects according to a sharding technique, wherein each respective data object of the plurality of data objects is stored as a respective plurality of shards, wherein each shard for the respective data object is stored on a different one of the plurality of storage devices than any other shard of the plurality of shards for the respective data object; and
a host-side network processing device, coupled between the network and a network connection to a host machine, and configured to:
receive, from the host machine, a data object and a request to store the data object;
send, to the storage system, a request for a plurality of storage locations of the storage system to store respective shards of the data object;
receive, from a particular server of the plurality servers, a location list including a plurality of location identifiers, wherein each location identifier corresponds to a respective storage location of the storage system;
determine an encoding matrix for the data object;
encode the data object into a plurality of shards based on the encoding matrix according to a sharding technique; and
generate a plurality of storage requests that each specify a different location identifier of the location list and that each includes a respective shard of the plurality of shards.

2. The system of claim 1 , wherein the host-side network processing device is further configured to send, to the storage system, the plurality of storage requests, and wherein at least two storage requests of the plurality of storage requests are sent to the storage system via the network by different network paths.

3. The system of claim 1 , wherein the host-side network processing device is further configured to:
receive, from respective servers of the plurality of servers in response to the plurality of storage requests, a plurality of respective storage responses that each confirm storage of a respective shard of the plurality of shards;
determine that a respective storage response of the plurality of storage responses has been received for each shard of the plurality of shards; and
send, to the storage system, a storage confirmation message that indicates that the respective storage responses have been received for each shard of the plurality of shards.

4. The system of claim 1 , wherein the location identifiers are encrypted such that the host-side network processing device is unable to decrypt the location identifiers.

5. The system of claim 1 , wherein the storage system is configured to:
receive, from the host-side network processing device, a different request for a different plurality of storage locations of the storage system to store different respective shards of a different data object;
determine that the storage system should encode the different data object into a different encoded data object; and
send, to the host-side network processing device, a request for the different data object.

6. A non-transitory, computer-readable storage medium storing program instructions that, when executed on one or more computers, cause the one or more computers to:
implement a server process configured to:
receive, from a host-side network processing device via a network coupled to the server, a request to store a data object as a plurality of shards;
generate a storage identifier for the data object and a location list that includes a plurality of location identifiers, wherein each location identifier specifies a different storage location for a respective shard of the plurality of shards;
send, in response to the request, the location list to the host-side network processing device;
receive, from the host-side network processing device, a particular storage request that includes a particular shard of the plurality of shards, wherein the particular storage request specifies a particular location identifier of the location list; and
send the particular shard and the storage identifier to be stored at a particular location of a particular storage device identified by the particular location identifier.

7. The non-transitory, computer-readable storage medium of claim 6 ,
wherein the server is part of a network-based storage service that comprises a plurality of storage devices,
wherein each location identifier on the location list corresponds to a different one of the plurality of storage devices,
wherein respective data objects are stored at the plurality of storage devices on behalf of a plurality of different clients, and
wherein at least some of the plurality of storage devices store shards for multiple different clients of the plurality of different clients.

8. The non-transitory, computer-readable storage medium of claim 6 ,
wherein the server is part of a network-based storage service that comprises a plurality of servers and a plurality of storage devices,
wherein the plurality of servers receives a plurality of storage requests each including a respective shard of the plurality of shards and a respective location identifier of the location list, and
wherein each of the plurality of storage requests is handled by a respective server selected based at least in part on a locality of the respective server to a respective storage device identified by the respective location identifier.

9. The non-transitory, computer-readable storage medium of claim 8 , wherein, in response to sending the location list, the server is further configured to:
receive, from the host-side network processing device, a different storage request that includes a different shard of the plurality of shards, wherein the different storage request specifies a different location identifier of the location list; and
send the different shard to a different location of a different storage device identified by the different location identifier.

10. The non-transitory, computer-readable storage medium of claim 6 ,
wherein, in response to receiving the request to store the data object, the server is further configured to send the storage identifier to the host-side network processing device, and
wherein the particular storage request includes the storage identifier.

11. The non-transitory, computer-readable storage medium of claim 6 , wherein, in response to the request, the server is further configured to:
determine object reconstruction information corresponding to the plurality of shards;
prior to receiving the particular shard, send the object reconstruction information to the host-side network processing device; and
send the object reconstruction information to a storage device.

12. The non-transitory, computer-readable storage medium of claim 6 , wherein, in response to the request, the server is further configured to:
send object reconstruction information corresponding to the plurality of shards to a storage device, wherein the request comprises the object reconstruction information.

13. The non-transitory, computer-readable storage medium of claim 6 , wherein in response to the request, the server is further configured to:
receive, from the host-side network processing device, an access control list; and
authenticate a request for the data object by comparing an identity of a requester corresponding to the request for the data object with the access control list.

14. The non-transitory, computer-readable storage medium of claim 6 , wherein, prior to returning the location list, the ser

Assignees

• Amazon Tech Inc

Inventors

• Sorenson Iii James Christopher

Classifications

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• H04L63/08

  for authentication of entities (cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32) · CPC title

• G06F2211/103

  Hybrid, i.e. RAID systems with parity comprising a mix of RAID types · CPC title

• H04L63/0428 Primary

  wherein the data content is protected, e.g. by encrypting or encapsulating the payload · CPC title

• G06F17/30091 Primary

  Physics · mapped topic