Snapshot and replication of a multi-stream application on multiple hosts at near-sync frequency

Patent metadata

Field	Value
Publication number	US-10073902-B2
Application number	US-201414495685-A
Country	US
Kind code	B2
Filing date	Sep 24, 2014
Priority date	Sep 24, 2014
Publication date	Sep 11, 2018
Grant date	Sep 11, 2018

Abstract

Storage access requests are received from one or more applications. Multiple servers update multiple virtual disks as directed by the storage access requests. The virtual disks store data that is write order dependent across the virtual disks. Logs are associated with the virtual disks. Information associated with each storage access request is stored in one of the logs. A cycle of log switching is performed. A write order consistent tracking coordinator coordinates the log switching with agents at the servers to maintain request ordering. Replication coordinators coordinate the application of the switched-out log files from primary storage to replica storage, creating a write-order consistent point on the replica side matching the primary side, and providing for failure resiliency regarding transfer of the logs. The replication logs may be received individually on the replica side from the servers on the primary side to enable highly scalable parallel/simultaneous transfers of the logs.

First claim

What is claimed is:

1. A method in a write order consistent tracking (WOCT) coordinator, comprising:
initiating a cycle of a log switching of a plurality of logs associated with a plurality of virtual disks at a plurality of computing devices, the initiating including taking an exclusive lock on each of a plurality of begin stage lock files, the virtual disks storing data that is write order dependent amongst the virtual disks, each computing device of the plurality of computing devices including at least one of
a virtual disk of the plurality of virtual disks that receives storage access requests from an application, the storage access requests including write requests, and
a log of the plurality of logs corresponding to the virtual disk that receives log queue entries corresponding to the storage access requests; and

coordinating the cycle of the log switching of the plurality of logs at the plurality of computing devices across the virtual disks to maintain request ordering for write order dependent requests.

2. The method of claim 1 , wherein said coordinating comprises:
enacting a plurality of stages to cause the switching of the plurality of logs at the plurality of computing devices.

3. The method of claim 2 , wherein
each begin stage lock file is associated with a corresponding stage of the plurality of stages; and
wherein said initiating comprises:
transmitting a log switching initiation instruction to a plurality of agents at the computing devices, each computing device including a corresponding agent of the plurality of agents.

4. The method of claim 3 , wherein said initiating further comprises:
receiving a response from each of the agents, each response received from an agent of the plurality agents indicating that the agent took a shared lock on each of a plurality of end stage lock files, each end stage lock file associated with a corresponding stage of the plurality of stages.

5. The method of claim 4 , wherein each stage of the plurality of stages is enacted by performing:
releasing the exclusive lock on the associated begin stage lock file to signal the beginning of the stage to the agents,
attempting to take an exclusive lock on the associated end stage lock file,
taking the exclusive lock on the associated end stage lock file when enabled by the agents having released all shared locks on the associated end stage lock file to signify completion of the stage by the agents, and
transitioning to enacting a next stage until a final stage of the plurality of stages is completed.

6. The method of claim 2 , wherein said coordinating comprises:
transmitting control codes and/or messages to a plurality of agents at the computing devices to enact the plurality of stages, each computing device including a corresponding agent of the plurality of agents.

7. The method of claim 6 , wherein each stage of the plurality of stages is enacted by performing:
transmitting a control code to the plurality of agents;
awaiting a response to the transmitted control code from each of the plurality of agents;
aborting the log switching if at least one of the agents does not respond with the awaited response within a predetermined time period for the plurality of stages to be completed; and
transitioning to enacting a next stage if all agents respond within the predetermined time period, said transitioning including completing the log switching when a final stage of the plurality of stages is completed.

8. A write order consistent tracking (WOCT) coordinator, comprising:
at least one processor circuit; and
memory that stores instructions for operations, the instructions defining:
a log switching initiator configured to communicate with a plurality of agents at a plurality of computing devices to initiate cycles of a log switching of a plurality of logs associated with a plurality of virtual disks at the plurality of computing devices and take an exclusive lock on each of a plurality of begin stage lock files, a cycle of the log switching including a switching out of each current log for a corresponding new log, each computing device of the plurality of computing devices including at least one of
a virtual disk of the plurality of virtual disks that receives storage access requests from an application, the storage access requests including write requests, and
a log of the plurality of logs corresponding to the virtual disk that receives log queue entries corresponding to the storage access requests; and

a log switching manager configured to coordinate the cycles of the log switching of the plurality of logs at the plurality of computing devices to maintain request ordering for write order dependent requests across virtual disks.

9. The WOCT coordinator of claim 8 , wherein the log switching manager is configured to enact a plurality of stages to cause a cycle of the switching of the plurality of logs at the plurality of computing devices.

10. The WOCT coordinator of claim 9 , wherein
each begin stage lock file is associated with a corresponding stage of the plurality of stages; and
wherein, for a cycle of the log switching, the log switching initiator is configured to:
transmit a log switching initiation instruction to the plurality of agents at the computing devices to initiate the log switching.

11. The WOCT coordinator of claim 10 , wherein the log switching initiator is configured to receive a response from each of the agents, each response received from an agent of the plurality agents indicating that the agent took a shared lock on each of a plurality of end stage lock files, each end stage lock file associated with a corresponding stage of the plurality of stages.

12. The WOCT coordinator of claim 11 , wherein to enact each stage of the plurality of stages, the log switching manager is configured to:
release the exclusive lock taken by the log switching initiator on the associated begin stage lock file to signal the beginning of the stage to the agents,
attempt to take an exclusive lock on the associated end stage lock file,
take the exclusive lock on the associated end stage lock file when enabled by the agents having released all shared locks on the associated end stage lock file to signify completion of the stage by the agents, and
transition to enacting a next stage until a final stage of the plurality of stages is completed.

13. A method in a write order consistent tracking (WOCT) coordinator, comprising:
initiating a cycle of a log switching of a plurality of logs associated with a plurality of virtual disks at a plurality of computing devices, the virtual disks storing data that is write order dependent amongst the virtual disks, each computing device of the plurality of computing devices including at least one of
a virtual disk of the plurality of virtual disks that receives storage access requests from an application, the storage access requests including write requests, and
a log of the plurality of logs corresponding to the virtual disk that receives log queue entries corresponding to the storage access requests; and

coordinating the cycle of the log switching of the plurality of logs at the plurality of computing devices across the virtual disks to maintain request ordering for write order dependent requests, said coordinating including enacting a plurality of stages to cause the switching of the plurality of logs at the plurality of computing devices, said enacting a plurality of stages comprising:
enacting a first stage during which a new log is initialized at each computing device of the plurality of computing devices;
enacting a second stage during which received log queue entries are blocked

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Kotagiri Sriravi
• Newaskar Rahul
• Bhave Ajay
• Joshi Yadnyesh

Classifications

• G06F2201/82

  Solving problems relating to consistency · CPC title

• G06F11/2094 Primary

  Redundant storage or storage space (G06F11/2056 takes precedence) · CPC title

• G06F11/2097

  maintaining the standby controller/processing unit updated (initialisation or re-synchronisation thereof G06F11/1658 and subgroups) · CPC title

• G06F2201/825

  the problem or solution involving locking · CPC title

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title