Appending to files via server-side chunking and manifest manipulation

Patent metadata

Field	Value
Publication number	US-9967298-B2
Application number	US-201514930571-A
Country	US
Kind code	B2
Filing date	Nov 2, 2015
Priority date	Mar 8, 2011
Publication date	May 8, 2018
Grant date	May 8, 2018

Abstract

The object storage system divides files into a number of object segments, each segment corresponding to a portion of the object, and stores each segment individually in the cloud storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. Manipulation of the individual object segments and/or the manifest is also used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

First claim

What is claimed is:

1. A network file system, the system comprising:
a storage management server, the storage management server further comprising a processor, a network interface and a filesystem export module; and
at least one computer-readable storage coupled to the storage management server, wherein the at least one computer-readable storage includes a plurality of individually individual stored object segments and a first manifest, wherein each of the plurality of individually stored object segments correspond to a portion of the data in the filesystem, and wherein the first manifest describes the relationship of the plurality of individually stored object segments to the filesystem sufficient for the filesystem export module to present a filesystem interface in response to communications received over the network interface,
wherein each of the plurality of individually stored object segments corresponds to a portion of data in an initial version of a segmented file in the filesystem, wherein a first individually stored object segment of the plurality of individually stored object segments is stored in a first storage pool, and wherein a partition is mapped to the first storage pool, the first storage pool is located in a first zone, and data stored within the first storage pool are subject, based on a single event, to a correlated loss of access,
wherein the storage management server calculates a first checksum for the first stored object segment, replicates the first individually stored object segment, and calculates a second checksum for the replicated object segment, wherein the first checksum is associated with the first stored object segment in the first manifest, and
wherein the storage management server stores the replicated object segment into a second storage pool different from the first storage pool upon determining the first checksum matches the second checksum, the second storage pool is located in a second zone different from the first zone, and data stored within the second storage pool are subject to a correlated loss of access.

2. The system of claim 1 , further comprising:
a snapshot module, wherein the snapshot module receives a request for write access to the filesystem and creates a second manifest responsive to the request.

3. The system of claim 2 , further comprising: a file writer module, wherein the file writer module stores at least one new
individually stored object segment containing the differential changes caused by the request for write access in a second storage pool, wherein the partition is mapped to the second storage pool, the second storage pool is located in a second zone different from the first zone, and data stored within the second storage pool are subject to a correlated loss of access.

4. The system of claim 3 , wherein the first manifest describes the relationship of the plurality of individually stored object segments to the segmented file to reconstitute the initial version of the segmented file.

5. The system of claim 4 , wherein the second manifest describes the relationship of individually stored object segments and the differential changes caused by the request for write access to the segmented file to reconstitute a modified version of the segmented file.

6. The system of claim 5 , wherein the first and second manifests are accessible in the filesystem.

7. The system of claim 1 , wherein the storage management server modifies the first manifest to identify a reconstruction of the segmented file.

8. The system of claim 7 , wherein the reconstruction includes the object data from the first data range as well as information from the plurality of preexisting object segments, and the reconstruction includes the replicated object segment rather than the first individually stored object segment.

9. The system of claim 1 , wherein the plurality of individual stored object segments corresponds to one or more blocks from a block device.

10. The system of claim 1 , wherein the plurality of individually stored object segments corresponds to one or more inodes from a filesystem.

11. The system of claim 1 , wherein the plurality of individually stored object segments corresponds to one or more files from a filesystem.

12. The system of claim 1 , wherein the network file system has a limit on a size of any single object that is uploaded.

13. The system of claim 12 , wherein each stored object segment is smaller than the limit.

14. The system of claim 12 , wherein the first manifest is separate from the plurality of individually stored object segments.

15. The system of claim 12 , wherein the first manifest is a header in the first stored object segment.

16. The system of claim 1 , wherein the first manifest references each of the plurality of individually stored object segments.

17. The system of claim 16 , wherein the first manifest references includes a name of each of the plurality of individually stored object segments.

Assignees

• Rackspace Us Inc

Inventors

• Barton Michael
• Reese Will
• Dickinson John A
• Payne Jay B
• Thier Charles B
• Holt Gregory

Classifications

• G06F11/1004

  to protect a block of data words, e.g. CRC or checksum (G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00) · CPC title

• G06F17/30153

  Physics · mapped topic

• G06F3/0619

  in relation to data integrity, e.g. data losses, bit errors · CPC title

• G06F17/30864

  Physics · mapped topic

• H04L65/403 Primary

  Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15) · CPC title