Method and apparatus for big data cloud storage resource management

Patent metadata

Field	Value
Publication number	US-2016357443-A1
Application number	US-201615174657-A
Country	US
Kind code	A1
Filing date	Jun 6, 2016
Priority date	Jun 5, 2015
Publication date	Dec 8, 2016
Grant date	—

Abstract

A system and method for storage resource management that improves the efficiency of big data cloud storage management using copy-on-write/read snapshots to redirect the I/O requests without moving the virtual disk is disclosed herein. The system includes storage pools, a virtualized server, and a centralized data center management server. The virtualized server includes a base disk and a virtualized disk having a copy-on-write and a copy-on-read snapshot process responsive to an I/O request that merges a previous snapshot stored on a first one of the plurality storage pools with the base disk stored on a second one of the plurality of storage pools, while it fetches a new snapshot to be stored on a third one of the plurality of storage pools from the previous snapshot or the base virtual disk. The system includes a performance model, employing an exponentially weighted moving average with an adjustable sliding window.

First claim

What is claimed is:

1 . A system for storage resource management, comprising:
a plurality of storage pools; at least one server, wherein each server of the at least one server comprises a corresponding processor of a corresponding at least one processor and a corresponding memory of a corresponding at least one memory, wherein each server of the at least one server is connected to the plurality of storage pools, wherein a virtualized server is implemented on a server of the at least one server, wherein the virtualized server has a plurality of base virtual disks, wherein the virtualized server has a plurality of snapshots on top of the plurality of base virtual disks such that each snapshot of the plurality of snapshots is on top of a corresponding base virtual disk of the plurality of base virtual disks, wherein the plurality of snapshots is a plurality of copy-on-write/read snapshots, wherein the plurality of base virtual disks and the plurality of copy-on-write/read snapshots are on the plurality of storage pools, wherein the virtualized server implements a copy-on-write/read snapshot creation process with respect to a specific base virtual disk stored on an initial storage pool of the plurality of storage pools and an existing snapshot stored in an old storage pool of the plurality of storage pools, such that the copy-on-write/read snapshot creation process creates a new snapshot on a new storage pool of the plurality of storage pools, wherein the existing snapshot corresponds to the specific base virtual disk stored on an initial storage pool, wherein the new storage pool is different from the old storage pool, wherein after creation of the new snapshot, (i) the existing snapshot becomes an old snapshot, (ii) the copy-on-write/read snapshot process initiates a merge, and (iii) the copy-on-write/read snapshot process initiates a fetch, such that until either the merge or the fetch is completed, the merge comprises merging data from the old snapshot stored on the old storage pool with the specific base virtual disk stored on the initial storage pool, wherein received write requests with respect to the specific base virtual disk are directed to the new snapshot and writes with respect to the specific base virtual disk are written to the new snapshot, and the fetch comprises received read requests with respect to the specific base virtual disk are directed to the new snapshot such that
(a) before a data block of the specific base disk is fetched to the new snapshot, in response to a read request with respect to the data block triggers the new snapshot to fetch latest data for the data block from either the old snapshot or the specific base virtual disk,
(b) after the data block is fetched to the new snapshot, further read requests with respect to the data block is read from the data block fetches to the new snapshot,
wherein after either the merge or the fetch is completed, and until both the merge and the fetch are completed, (i) if the merge is completed and the fetch is not completed, the merge terminates and the fetch continues, (ii) if the fetch is completed and the merge is not completed, the fetch terminates and the merge continues, wherein after both the merge and the fetch are completed, write requests and read requests with respect to the specific base virtual disk are sent to the new snapshot and are completely run on the new storage pool.

2 . The system according to claim 1 , wherein each base virtual disk of the plurality of base virtual disks corresponds to a virtual machine of a plurality of virtual machines, wherein the at least one server comprises:
a centralized data center management server, wherein a storage manager is implemented on the centralized data center management server, wherein the storage manager monitors IO characteristics of the plurality of virtual machines, wherein the centralized data center management server is connected to each other server of the at least one server and is coupled to the virtualized server, wherein the storage manager provides instructions to the virtualized server regarding imitating the copy-on-write/read snapshot process, including identification of the new storage pool.

3 . The system according to claim 2 , wherein the storage manager implements performance tracking and performance load balancing based on the monitored IO characteristics of the plurality of virtual machines.

4 . The system of claim 3 , wherein the storage manager implements capacity prediction and capacity planning based on the monitored IO characteristics of the plurality of virtual machines.

5 . The system according to claim 3 , wherein performance tracking tracks storage latency using dynamic sliding window tracking that samples the latency of each storage pool for q times and uses a latency prediction method to predict the storage latency.

6 . The system of claim 5 , wherein the latency prediction method is an exponentially weighted moving average (EWMA) method.

7 . The system of claim 5 , wherein the latency prediction method is an exponentially smoothing method.

8 . The system of claim 5 , wherein the latency prediction method is an auto regression method.

9 . The system according to claim 1 , wherein each snapshot of the plurality of snapshots holds a corresponding working set of running applications with respect to the corresponding base virtual disk.

10 . A method of storage resource management, comprising:
providing a plurality of storage pools, providing at least one server, wherein each server of the at least one server comprises a corresponding processor of a corresponding at least one processor and a corresponding memory of a corresponding at least one memory, wherein each server of the at least one server is connected to the plurality of storage pools, implementing a virtual server on a first server of the at least one server, wherein the virtual server has a plurality of base virtual disks, wherein the virtualized server has a plurality of snapshots on top of the plurality of base virtual disks such that each snapshot of the plurality of snapshots is on top of a corresponding base virtual disk of the plurality of base virtual disks, wherein the plurality of snapshots is a plurality of copy-on-write/read snapshots, storing the plurality of base disks and the plurality of snapshots on the plurality of storage pools, implementing a copy-on-write/read a snapshot wherein the virtualized server implements a copy-on-write/read snapshot creation process with respect to a specific base virtual disk stored on an initial storage pool of the plurality of storage pools and an existing snapshot stored in an old storage pool of the plurality of storage pools, such that the copy-on-write/read snapshot creation process creates a new snapshot on a new storage pool of the plurality of storage pools, wherein the existing snapshot corresponds to the specific base virtual disk stored on an initial storage pool, wherein the new storage pool is different from the old storage pool, wherein after creation of the new snapshot, (i) the existing snapshot becomes an old snapshot, (ii) the copy-on-write/read snapshot process initiates a merge, wherein the merge comprises merging data from the old snapshot stored on the old storage pool with the specific base virtual disk stored on the initial storage pool, wherein write requests received are directed to the new snapshot with respect to the specific base virtual disk, (iii) the copy-on-write/read snapshot process initiates a fetch, where the fetch comprises sending read requests to the new snapshot such that (i) before a data block of the specific base disk is fetched to the new snapshot, in response to a read

Assignees

• Univ Florida

Inventors

• Li Tao
• Zhou Ruijin

Classifications

• G06F3/065 Primary

  Replication mechanisms · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0659

  Command handling arrangements, e.g. command buffers, queues, command scheduling · CPC title

• G06F3/0647

  Migration mechanisms · CPC title

• G06F3/0619

  in relation to data integrity, e.g. data losses, bit errors · CPC title