Object storage system with priority meta object replication

Patent metadata

Field	Value
Publication number	US-11079960-B2
Application number	US-201916447321-A
Country	US
Kind code	B2
Filing date	Jun 20, 2019
Priority date	Jun 20, 2019
Publication date	Aug 3, 2021
Grant date	Aug 3, 2021

Abstract

Example object storage systems, replication managers, and methods provide meta objects for priority replication of configuration data between data object stores. A meta object may be generated that includes configuration data. Data objects are selected for replication from a first replication queue. Meta objects associated with the data objects in the first replication queue are replicated, independent of the first replication queue, between object data stores.

First claim

What is claimed is:

1. A system, comprising:
a first object data store configured to:
store user data objects, wherein each user data object includes user object data stored in a storage medium of the first object data store; and
associate, in a first metadata store, a storage location for the user object data for each user data object with an object identifier and object metadata for that user data object;

the first metadata store configured to:
store the object identifiers and object metadata for accessing user data objects stored in the first object data store; and
store configuration data for the first object data store, wherein the configuration data
includes configuration parameters for rebuilding the configuration data in
a recovery metadata store;

a meta object generator configured to:
generate a meta object, wherein:
the meta object is a data object containing object data; and
the object data includes meta object data selected from the configuration parameters;

assign a meta object identifier to the meta object;
store the meta object with a related set of data objects in the first object data store; and
associate, in the first metadata store, a storage location for the meta object data with the meta object identifier and object metadata for the meta object;

a replication manager configured to:
select data objects from a first replication queue;
replicate the selected data objects from the first object data store to a second object data store; and
replicate, responsive to the meta object generator generating the meta object and with priority over the first replication queue, the meta object from the first object data store to the second object data store; and

a storage interface configured to:
read the configuration data from the first metadata store to define a storage protocol for processing object data requests for the related set of data objects stored in the first object data store; and
read, responsive to a failure of the first metadata store, rebuilt configuration data from the recovery metadata store, wherein the rebuilt configuration data is based on the configuration parameters from the meta object.

2. The system of claim 1 , wherein:
the first object data store is configured to store a plurality of data objects in a data object bucket;
the meta object data includes bucket configuration parameters; and
the selected data objects includes user data objects stored in the data object bucket.

3. The system of claim 1 , wherein:
the storage interface is further configured to:
receive a configuration change for updated configuration data for the first object data store;
responsive to receiving the configuration change, generate a meta object work item; and
send the meta object work item to the replication manager, wherein replicating the meta object is based on the meta object work item.

4. The system of claim 1 , wherein the replication manager is further configured to:
add the meta object to a second replication queue; and
select, responsive to adding the meta object to the second replication queue, the meta object from the second replication queue for replication before selecting data objects in the first replication queue for replication.

5. The system of claim 1 , further comprising:
a first replication engine configured to replicate, responsive to the replication manager,
the selected data objects from the first replication queue; and
a priority replication engine configured to replicate, responsive to the replication manager, the meta object.

6. The system of claim 5 , wherein the priority replication engine includes at least one dedicated processing thread for priority processing of meta object replication.

7. The system of claim 1 , wherein:
the storage interface is further configured to:
receive a configuration change for updated configuration data for the first object data store;
initiate, responsive to the configuration change, the meta object generator to generate an updated meta object; and
initiate, responsive to the configuration change, the replication manager to replicate the updated meta object from the first object data store to the second object data store.

8. The system of claim 7 , wherein the replication manager is configured to execute, responsive to the configuration change, a put operation of the updated meta object to the second object data store to replicate the updated meta object.

9. The system of claim 1 , further comprising:
a scheduler configured to:
select a predetermined interval for meta object updates, wherein the meta object generator is further configured to periodically generate, at the predetermined interval, updated meta objects including changed configuration data.

10. A computer-implemented method, comprising:
storing, in a first object data store, user data objects, wherein each user data object includes user object data stored in a storage medium of the first object data store;
storing, in a first metadata store, a storage location for the user object data for each user data object with an object identifier and object metadata for that user data object;
storing, in the first metadata store, configuration data for the first object data store, wherein the configuration data-includes configuration parameters for rebuilding the configuration data in a recovery metadata store;
generating a meta object, wherein:
the meta object is a data object containing object data; and
the object data includes meta object data selected from configuration data for the first object data store;

assigning a meta object identifier to the meta object;
storing the meta object with a related set of data objects in the first object data store;
associating, in the first metadata store, a storage location for the meta object data with the meta object identifier and object metadata for the meta object;
selecting, using a first replication queue, data objects from the first object data store;
replicating selected data objects from the first object data store to a second object data store;
replicating, responsive to generating the meta object and with priority over the first replication queue, the meta object from the first object data store to the second object data store;
reading the configuration data from the first metadata store to define a storage protocol for processing object data requests for the related set of data objects stored in the first object data store; and
reading, responsive to a failure of the first metadata store, rebuilt configuration data from the recovery metadata store, wherein the rebuilt configuration data is based on the configuration parameters from the meta object.

11. The computer-implemented method of claim 10 , further comprising:
configuring the first object data store to store a plurality of user data objects and the meta object in a data object bucket, wherein:
the meta object data includes bucket configuration data for the data object bucket; and
the selected data objects includes the plurality of user data objects stored in the data object bucket.

12. The computer-implemented method of claim 10 , further comprising:
intercepting a configuration change for updated configuration data for the first object data store; and
generating, responsive to intercepting the configuration change, a meta object work item, wherein replicating the meta object is based on the meta object work item.

13. The computer-implemented method of claim 10 , further comprising:
adding the meta object to a second replication queue; and
selecting, responsive to adding the meta object to the second replication queue,

Assignees

• Western Digital Tech Inc

Inventors

• Cheru Tomy Ammuthan
• D'Halluin Carl
• Pyati Ameet

Classifications

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0644

  Management of space entities, e.g. partitions, extents, pools · CPC title

• G06F3/0614

  Improving the reliability of storage systems · CPC title

• G06F3/0683

  Plurality of storage devices · CPC title

• G06F3/065 Primary

  Replication mechanisms · CPC title