Issuing efficient writes to erasure coded objects in a distributed storage system via adaptive logging

Patent metadata

Field	Value
Publication number	US-11467746-B2
Application number	US-202017089605-A
Country	US
Kind code	B2
Filing date	Nov 4, 2020
Priority date	Apr 7, 2020
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

Techniques for issuing efficient writes to an erasure coded storage object in a distributed storage system via adaptive logging are provided. In one set of embodiments, a node of the system can receive a write request for updating one or more logical data blocks of the storage object and determine whether a size of the one or more logical data blocks meets or exceeds a threshold size. Upon determining that the size of the one or more logical data blocks meets or exceeds the threshold size, the node can allocate a segment in a capacity object of the storage object, write the one or more logical data blocks via a full stripe write to the segment, and write metadata for the one or more logical data blocks to a log record in a log of a metadata object of the storage object. The metadata written to the log record can include mappings between logical block addresses (LBAs) of the one or more logical data blocks and physical block addresses (PBAs) where the one or more logical data blocks reside in the segment.

First claim

What is claimed is:

1. A method for issuing efficient writes to an erasure coded storage object maintained by a distributed storage system via adaptive logging, the method comprising:
receiving, by a node of the distributed storage system, a write request for updating one or more logical data blocks of the storage object;
determining, by the node, whether a size of the one or more logical data blocks meets or exceeds a threshold size; and
upon determining that the size of the one or more logical data blocks meets or exceeds the threshold size:
allocating, by the node, a segment in a capacity object of the storage object;
writing, by the node, the one or more logical data blocks via a full stripe write to the segment; and
writing, by the node, metadata for the one or more logical data blocks to a log record in a log of a metadata object of the storage object, the metadata include mappings between logical block addresses (LBAs) of the one or more logical data blocks and physical block addresses (PBAs) where the one or more logical data blocks reside in the segment.

2. The method of claim 1 wherein the capacity object is striped across a storage tier of the distributed storage system in accordance with an erasure coding scheme assigned to the storage object, and wherein the threshold size is based on a full stripe size of the capacity object.

3. The method of claim 1 wherein the threshold size is equal to a size of the segment.

4. The method of claim 1 wherein the log record excludes data of the one or more logical data blocks.

5. The method of claim 1 wherein the capacity object is striped across a first storage tier of the distributed storage system in accordance with an erasure coding scheme assigned to the storage object, and wherein the metadata object is mirrored across a second storage tier of the distributed storage system in a manner that enables the metadata object to achieve an equivalent level of fault tolerance as the capacity object.

6. The method of claim 1 wherein the metadata object is created and managed using a overwrite-based file system disk layout, and wherein the capacity object is created and managed using a log-structured file system (LFS) disk layout.

7. The method of claim 1 further comprising, upon determining that the size of the one or more logical data blocks does not meet or exceed the threshold size:
writing data and metadata for the one or more logical data blocks to another log record in the log;
placing data for the one or more logical data blocks in one or more free slots of an in-memory bank, the in-memory bank being configured to hold a predefined number of stripes of the storage object in accordance with an erasure coding scheme assigned to the storage object;
in response to the placing, determining whether the in-memory bank has become full; and
if the in-memory bank has become full:
computing and filling one or more parity blocks for each stripe of the storage object in the in-memory bank;
allocating another segment in the capacity object for holding contents of the in-memory bank; and
writing contents of the in-memory bank via a full stripe write to said another segment.

8. A non-transitory computer readable storage medium having stored thereon program code executable by a node in a distributed storage system, the program code embodying a method for issuing efficient writes to an erasure coded storage object maintained by the distributed storage system via adaptive logging, the method comprising:
receiving a write request for updating one or more logical data blocks of the storage object;
determining whether a size of the one or more logical data blocks meets or exceeds a threshold size; and
upon determining that the size of the one or more logical data blocks meets or exceeds the threshold size:
allocating a segment in a capacity object of the storage object;
writing the one or more logical data blocks via a full stripe write to the segment; and
writing metadata for the one or more logical data blocks to a log record in a log of a metadata object of the storage object, the metadata include mappings between logical block addresses (LBAs) of the one or more logical data blocks and physical block addresses (PBAs) where the one or more logical data blocks reside in the segment.

9. The non-transitory computer readable storage medium of claim 8 wherein the capacity object is striped across a storage tier of the distributed storage system in accordance with an erasure coding scheme assigned to the storage object, and wherein the threshold size is based on a full stripe size of the capacity object.

10. The non-transitory computer readable storage medium of claim 8 wherein the threshold size is equal to a size of the segment.

11. The non-transitory computer readable storage medium of claim 8 wherein the log record excludes data of the one or more logical data blocks.

12. The non-transitory computer readable storage medium of claim 8 wherein the capacity object is striped across a first storage tier of the distributed storage system in accordance with an erasure coding scheme assigned to the storage object, and wherein the metadata object is mirrored across a second storage tier of the distributed storage system in a manner that enables the metadata object to achieve an equivalent level of fault tolerance as the capacity object.

13. The non-transitory computer readable storage medium of claim 8 wherein the metadata object is created and managed using a overwrite-based file system disk layout, and wherein the capacity object is created and managed using a log-structured file system (LFS) disk layout.

14. The non-transitory computer readable storage medium of claim 8 wherein the method further comprises, upon determining that the size of the one or more logical data blocks does not meet or exceed the threshold size:
writing data and metadata for the one or more logical data blocks to another log record in the log;
placing data for the one or more logical data blocks in one or more free slots of an in-memory bank, the in-memory bank being configured to hold a predefined number of stripes of the storage object in accordance with an erasure coding scheme assigned to the storage object;
in response to the placing, determining whether the in-memory bank has become full; and
if the in-memory bank has become full:
computing and filling one or more parity blocks for each stripe of the storage object in the in-memory bank;
allocating another segment in the capacity object for holding contents of the in-memory bank; and
writing contents of the in-memory bank via a full stripe write to said another segment.

15. A computer system acting as a node in a distributed storage system, the computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
receive a write request for updating one or more logical data blocks of the storage object;
determine whether a size of the one or more logical data blocks meets or exceeds a threshold size; and

upon determining that the size of the one or more logical data blocks meets or exceeds the threshold size:
allocate a segment in a capacity object of the storage object; write the one or more logical data blocks via a full stripe write to the segment; and write metadata for the one or more logical data blocks to a log record in a log of a metadata object of the storage object, the metadata include mappings between logical block addresses (LBAs) of the one or more logical data blocks and physical block addresse

Assignees

• Vmware Inc

Inventors

• Knauft Eric
• Wang Wenguang
• Renauld Pascal

Classifications

• G06F11/076

  by exceeding a count or rate limit, e.g. word- or bit count limit · CPC title

• G06F3/0631

  by allocating resources to storage systems · CPC title

• G06F3/061

  Improving I/O performance · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0619

  in relation to data integrity, e.g. data losses, bit errors · CPC title