Using erasure coding across multiple regions to reduce the likelihood of losing objects maintained in cloud object storage

Patent metadata

Field	Value
Publication number	US-11544147-B2
Application number	US-202016882338-A
Country	US
Kind code	B2
Filing date	May 22, 2020
Priority date	May 22, 2020
Publication date	Jan 3, 2023
Grant date	Jan 3, 2023

Abstract

Techniques for using erasure coding across multiple regions to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload each of a plurality of data objects to each of a plurality of regions of the cloud object storage platform. The computer system can further compute a parity object based on the plurality of data objects, where the parity object encodes parity information for the plurality of data objects. The computer system can then upload the parity object to another region of the cloud object storage platform different from the plurality of regions.

First claim

What is claimed is:

1. A method comprising:
uploading, by a computer system, each of a plurality of data objects to each of a plurality of regions of a cloud object storage platform;
computing, by the computer system, a parity object based on the plurality of data objects, the parity object encoding parity information for the plurality of data objects; and
uploading, by the computer system, the parity object to another region of the cloud object storage platform different from the plurality of regions,
wherein metadata for the plurality of data objects is maintained in a metadata database that resides solely at a first region of the cloud object storage platform, and
wherein the plurality of data objects are garbage collected in a manner that avoids transferring data content of the plurality of data objects across the plurality of regions.

2. The method of claim 1 further comprising:
attempting to read a data object in the plurality of data objects from the region of the cloud object storage platform to which the data object was uploaded; and
if the read attempt fails:
reading other data objects in the plurality of data objects from the regions to which the other data objects was uploaded;
reading the parity object from said another region;
reconstructing the data object based on the other data objects and the parity object.

3. The method of claim 1 wherein object failures within each of the plurality of regions are correlated.

4. The method of claim 1 wherein object failures across the plurality of regions are independent.

5. The method of claim 1 wherein a master garbage collector component runs at the first region, and
wherein the master garbage collector component:
identifies, by accessing the metadata database, at least one data object in the plurality of data objects as a candidate for garbage collection; and
transmits, to a slave garbage collector component running at a second region where the at least one data object was uploaded, a command to garbage collect the candidate data object.

6. The method of claim 5 wherein the master garbage collector component further:
receives, from the slave garbage collector component, a list of logical data block numbers included in the candidate data object;
determines, by accessing the metadata database, a range of live data blocks in the candidate data object;
transmits the range of live data blocks to the slave garbage collector component;
receives a garbage collection acknowledgment from the slave garbage collector component; and
in response to receiving the garbage collection acknowledgment, causes the candidate data object to be deleted.

7. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code embodying a method comprising:
uploading each of a plurality of data objects to each of a plurality of regions of a cloud object storage platform;
computing a parity object based on the plurality of data objects, the parity object encoding parity information for the plurality of data objects; and
uploading the parity object to another region of the cloud object storage platform different from the plurality of regions,
wherein metadata for the plurality of data objects is maintained in a metadata database that resides solely at a first region of the cloud object storage platform, and
wherein the plurality of data objects are garbage collected in a manner that avoids transferring data content of the plurality of data objects across the plurality of regions.

8. The non-transitory computer readable storage medium of claim 7 wherein the method further comprises:
attempting to read a data object in the plurality of data objects from the region of the cloud object storage platform to which the data object was uploaded; and
if the read attempt fails:
reading other data objects in the plurality of data objects from the regions to which the other data objects was uploaded;
reading the parity object from said another region;
reconstructing the data object based on the other data objects and the parity object.

9. The non-transitory computer readable storage medium of claim 7 wherein object failures within each of the plurality of regions are correlated.

10. The non-transitory computer readable storage medium of claim 7 wherein object failures across the plurality of regions are independent.

11. The non-transitory computer readable storage medium of claim 7 wherein a master garbage collector component runs at the first region, and
wherein the master garbage collector component:
identifies, by accessing the metadata database, at least one data object in the plurality of data objects as a candidate for garbage collection; and
transmits, to a slave garbage collector component running at a second region where the at least one data object was uploaded, a command to garbage collect the candidate data object.

12. The non-transitory computer readable storage medium of claim 11 wherein the master garbage collector component further:
receives, from the slave garbage collector component, a list of logical data block numbers included in the candidate data object;
determines, by accessing the metadata database, a range of live data blocks in the candidate data object;
transmits the range of live data blocks to the slave garbage collector component;
receives a garbage collection acknowledgment from the slave garbage collector component; and
in response to receiving the garbage collection acknowledgment, causes the candidate data object to be deleted.

13. A computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
upload each of a plurality of data objects to each of a plurality of regions of a cloud object storage platform;
compute a parity object based on the plurality of data objects, the parity object encoding parity information for the plurality of data objects; and
upload the parity object to another region of the cloud object storage platform different from the plurality of regions,

wherein metadata for the plurality of data objects is maintained in a metadata database that resides solely at a first region of the cloud object storage platform, and
wherein the plurality of data objects are garbage collected in a manner that avoids transferring data content of the plurality of data objects across the plurality of regions.

14. The computer system of claim 13 wherein the program code further causes the processor to:
attempt to read a data object in the plurality of data objects from the region of the cloud object storage platform to which the data object was uploaded; and
if the read attempt fails:
read other data objects in the plurality of data objects from the regions to which the other data objects was uploaded;
read the parity object from said another region;
reconstruct the data object based on the other data objects and the parity object.

15. The computer system of claim 13 wherein object failures within each of the plurality of regions are correlated.

16. The computer system of claim 13 wherein object failures across the plurality of regions are independent.

17. The computer system of claim 13 wherein a master garbage collector component runs at the first region, and
wherein the master garbage collector component:
identifies, by accessing the metadata database, at least one data object in the plurality of data objects as a candidate for garbage collection; and
transmits,

Assignees

• Vmware Inc

Inventors

• Wang Wenguang
• Gao Junlong
• Gunturu Vamsi

Classifications

• G06F11/1088 Primary

  Reconstruction on already foreseen single or plurality of spare disks · CPC title

• G06F16/162

  Delete operations (erasing in storage systems G06F3/0652) · CPC title

• G06F16/164

  File meta data generation · CPC title

• G06F11/076

  by exceeding a count or rate limit, e.g. word- or bit count limit · CPC title

• G06F12/0253

  Garbage collection, i.e. reclamation of unreferenced memory · CPC title