Distributed data object management system operations

Patent metadata

Field	Value
Publication number	US-11003532-B2
Application number	US-201715626073-A
Country	US
Kind code	B2
Filing date	Jun 16, 2017
Priority date	Jun 16, 2017
Publication date	May 11, 2021
Grant date	May 11, 2021

Abstract

In various embodiments, methods and systems for implementing distributed data object management are provided. The distributed data object management system includes a local metadata-consensus information store and one or more remote metadata-consensus information stores for metadata-consensus information and a local data store and one or more remote data stores for erasure coded fragments. For a write operation, corresponding metadata writes and data writes are performed in parallel using a metadata write path and a data write path, respectively, when writing to the local metadata-consensus information store and the one or more remote metadata-consensus information stores and the local data store and the one or more remote data stores. And, for a read operation, corresponding metadata reads and data reads are performed in parallel using a metadata read path and a data read path, respectively, when reading from the metadata-consensus information stores and the data stores.

First claim

The invention claimed is:

1. A system for implementing distributed data object management, the system comprising:
one or more computer processors; and
computer memory storing computer-usable instructions that, when used by the one or more computer processors, cause the one or more computers processors to perform, in a distributed data object management system, operations comprising:
applying a consensus protocol that comprises instructions for consensus operations, wherein the instructions are based on a fast path protocol and a slow path protocol of the consensus protocol, the instructions include each of the following:
write common-case consensus operations of the fast path protocol;
read common-case consensus operations of the fast path protocol;
write contention-case consensus operations of the slow path protocol associated with failed write common-case consensus operations of the fast path protocol;
read contention-case consensus operations of the slow path protocol associated with failed read common-case consensus operations of the fast path protocol;
determining whether a write operation or a read operation is for a common case or a contention-case;
based on a consensus protocol and the common-case or the contention-case, executing the write operation and the read operation,
wherein the consensus operations of the consensus protocol are executed using a metadata write path and a data write path, a metadata read path and the data read path;
the metadata write path and the data write path, and the metadata read path and the data read path each in combination are used in parallel based on the instructions in the consensus protocol to respectively write and read to each of the following:
local metadata-consensus information stores, remote metadata-consensus information stores, local data stores, and remote data stores, for executing the write operation and the read operation,
wherein the write operation is executed based on a metadata write path and a data write path, wherein writing using the metadata write path and the data write path is performed in parallel, and
wherein the read operation is executed based on a metadata read path and a data read path, wherein reading using the metadata read path and the data read path is performed in parallel.

2. The system of claim 1 , wherein the distributed data object management system is developed based on a distributed storage system comprising:
a local metadata-consensus information store and one or more remote metadata-consensus information stores,
wherein a metadata-consensus information store is a table store for metadata-consensus information, the metadata-consensus information corresponds to erasure coded fragments of the data object; and
a local data store and one or more remote data stores, wherein a data store is an object store that stores the erasure coded fragments of the data object, the local data store and the one or more remote data stores store the erasure coded fragments of the data object that correspond to metadata-consensus information in the local metadata-consensus information store and the one or more remote metadata-consensus information stores.

3. The system of claim 2 , wherein executing the write operation comprises:
writing, to the local metadata-consensus information store and the one or more remote metadata-consensus information stores using the metadata write path, metadata-consensus information of erasure coded fragments of a data object,
wherein the metadata-consensus information corresponds to erasure coded fragments of the data object; and
writing, to a local data store and one or more remote data stores, wherein metadata-consensus information for an erasure coded fragment comprises a known committed version identifier, wherein the known committed version identifier is accessible for executing the write operation based on the consensus protocol; and
wherein executing the read operation comprises:
reading, from the local metadata-consensus information store and the one or more remote metadata-consensus information stores, using the metadata read path, metadata-consensus information of the erasure coded fragments of the data object, and reading, from the local data store or the one or more remote data stores, using the data read path, the erasure coded fragments of the data object.

4. The system of claim 1 , wherein executing the write operation for the common-case comprises:
determining a next version number to execute the write operation using the consensus protocol, wherein the next version number is determined based on a known committed version number at the local metadata-consensus information store; and
initiating a fast-round write using the fast-path protocol of the consensus protocol to update a local metadata store and one or more remote metadata-consensus information stores,
wherein the fast-path protocol comprises a pre-accept request of the fast-round write for updating the local metadata store and the one or more remote metadata-consensus information store.

5. The system of claim 1 , wherein executing the write operation for the contention-case comprises:
determining a next version number to execute the write operation using the consensus protocol, wherein the next version number is determined based on a known committed version number at a local metadata-consensus information store;
initiating a fast-round write using the fast-path protocol of the consensus protocol to update the local metadata store and one or more remote metadata-consensus information store,
wherein the fast-path protocol comprises a pre-accept request of the fast-round write for updating the local metadata store and the one or more remote metadata-consensus information stores;
determining that the fast-round write failed based on the pre-accept request not receiving quorum of acceptances for the fast-round write; and
initiating a classic-round write using the slow-path protocol of the consensus protocol to update the local metadata-consensus information store and the one or more remote metadata-consensus information store,
wherein the slow-path protocol comprises both a prepare request and an accept request of the classic-round write for updating the local metadata store and the one or more remote metadata-consensus information stores.

6. The system of claim 1 , wherein executing the read operation for the common-case comprises:
initiating a fast-round read using the fast-path protocol of the consensus protocol, wherein the fast-path protocol comprises reading from each of the local metadata stores and one or more remote metadata-consensus information stores; and
returning a matching known committed version, wherein the matching known committed version is matched and stored at a majority of the local metadata stores and the one or more remote metadata-consensus information stores; and
wherein a next version number has not been accepted at any of a local metadata store or the one or more remote metadata-consensus information stores.

7. The system of claim 1 , wherein executing the read operation for the contention-case comprises:
initiating a fast-round read using the fast-path protocol of the consensus protocol, wherein the fast-path protocol comprises reading from each of the local metadata stores and one or more remote metadata-consensus information stores;
determining that a next version number has been accepted at least one of the local metadata store and the one or more remote metadata-consensus information stores; and
initiating a classic-round read using the slow-path protocol of the consensus protocol, wherein the slow-path protocol comprises determining whether the next version number is a correct version number for each of the local metadata store and the one

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Huang Cheng
• Li Jin
• Ogus Aaron William
• Phillips Douglas W
• Chen Yu Lin
• Mu shuai
• Li Jinyang

Classifications

• H03M13/373 Primary

  with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes · CPC title

• H03M13/3761

  using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes · CPC title

• H03M13/1515

  Reed-Solomon codes · CPC title

• G06F11/1076 Primary

  Parity data used in redundant arrays of independent storages, e.g. in RAID systems · CPC title

• H03M13/154

  Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial · CPC title