Reclaiming space from file system hosting many primary storage objects and their snapshots

Patent metadata

Field	Value
Publication number	US-9400741-B1
Application number	US-201414319455-A
Country	US
Kind code	B1
Filing date	Jun 30, 2014
Priority date	Jun 30, 2014
Publication date	Jul 26, 2016
Grant date	Jul 26, 2016

Abstract

Reclaiming storage from a file system hosting of storage objects includes assigning objects and respective files to version sets, each version set containing a primary object and respective snapshots. Each block of storage is associated with the version set of the file(s) that contain the block. Blocks residing in an extent that is to be reclaimed are identified, and for each block (i) an offset is obtained that specifies a location of the block in a file, (ii) files whose objects are assigned to the version set associated with the block are identified, (iii) a block pointer at the location in each of the identified underlying files is examined, and for each file whose block pointer points to the block, updating the block pointer to point to a new block location outside of the extent, to which the block is moved.

First claim

What is claimed is:

1. A method of operating a storage processor to reclaim storage assigned to an upper file system hosting a set of storage objects including primary objects and respective snapshot objects, the storage objects hosted as respective files of the upper file system backed by an underlying file of a lower file system, the underlying file constituted by respective sets of fixed-size blocks of the storage, the method comprising:
assigning the storage objects and their respective files to version sets, each primary object and its respective snapshot objects being assigned to a respective distinct version set, the files of each version set selectively sharing blocks, each block having corresponding block-specific block metadata containing a respective version set identifier identifying the version set to which the files constituted by the block are assigned, the blocks of a given file having respective distinct block-specific block metadata, the block metadata forming part of the upper file system;
identifying blocks in an extent of the underlying file to be moves, and for each identified block:
i. obtaining an offset specifying a block location of the block in a file of the version set;
ii. obtaining the version set identifier from the block metadata for the block and using the version set identifier to identify the files whose corresponding storage objects are assigned to the version set with which the block is associated;
iii. using the offset to examine a block pointer at the block location in each of the identified files, and for each file whose block pointer points to the block, updating the block pointer to point to a new block location outside of the extent; and
iv. mobbing the block to the new block location; and

upon completion of the above for all blocks to be moved, removing the extent from the underlying file.

2. A method according to claim 1 , wherein assigning the storage objects and their respective files to version sets includes recording, in a version set database, the version set identifiers in association with file identifiers of files, the version set identifiers identifying the respective version sets, the file identifiers for each version set identifier identifying the files assigned to the respective version set, and wherein using the version set identifier to identify the files whose corresponding storage objects are assigned to the version set with which the block is associated includes applying the version set identifier for the block to the version set database and retrieving the corresponding file identifiers.

3. A method according to claim 1 , wherein the offset for each block is included in the respective block metadata for the block.

4. A method according to claim 1 , wherein the upper file system is a thin file system to which underlying storage is automatically assigned by the storage processor, and wherein the method further includes continually monitoring an amount of free space being unused in storage assigned to the upper file system, and wherein the steps for reclaiming storage are initiated automatically by the storage processor in response to the amount of free space increasing above a predetermined threshold.

5. A method according to claim 1 , wherein storage is allocated to the upper file system in multi-block slices each including a predetermined number of logically contiguous blocks, and where removing the extent from the underlying file includes removing the slices populating the extent.

6. A method according to claim 5 , wherein the slices are drawn from and returned to a pool indirectly obtained from an underlying logical unit of storage defined on physical storage devices.

7. A method according to claim 5 , wherein blocks of a given slice in the extent are data blocks and are moved to respective distinct new locations in respective different slices remaining assigned to the underlying file.

8. A method according to claim 7 , wherein the method is used for reclaiming storage only in the form of user data as opposed to file metadata, the user data stored in blocks of data slices, the file metadata being stored on distinct metadata slices that are reclaimed by a separate process moving entire metadata slices intact.

9. A method according to claim 1 , further including grouping the blocks by version set identifier and moving a first group of blocks for a first version set before moving a second group of blocks for a second version set, and wherein examining block pointers in the files for each group of blocks includes examining all block pointers for the group of blocks in a first file of the first version set before examining all block pointers for the group of blocks in a second file of the first version set.

10. A storage processor operative to reclaim storage assigned to an upper file system hosting a set of storage objects including primary objects and respective snapshot objects, the storage objects hosted as respective files of the upper file system backed by an underlying file of a lower file system, the underlying files constituted by respective sets of fixed-size blocks of the storage, the storage processor comprising:
communications interface circuitry connecting the storage processor to one or more host computers and to underlying physical storage on which the storage objects are stored, the host computers accessing the storage objects using respective storage commands directed to the storage processor;
one or more processors; and
memory storing computer program instructions that are executed by the processors to cause the storage processor to reclaim the storage by:
assigning the storage objects and their respective files to version sets, each primary object and its respective snapshot objects being assigned to a respective distinct version set, the files of each version set selectively sharing blocks, each block having corresponding block-specific block metadata containing a respective version set identifier identify the version set to which the files constituted by the block are assigned, the blocks of a given file having respective distinct block-specific block metadata, the block metadata forming part of the upper file system;
identifying blocks in an extent of the underlying file to be moved, and for each identified block:
i. obtaining an offset specifying a block location of the block in a file of the version set;
ii. obtaining the version set identifier from the block metadata for the block and using the version set identifier to identify the files whose corresponding storage objects are assigned to the version set with which the block is associated;
iii. using the offset to examine a block pointer at the block location in each of the identified files, and for each file whose block pointer points to the block, updating the block pointer to point to a new block location outside of the extent; and
iv. moving the block to the new block location; and

upon completion of the above for all blocks to be moved, remoting the extent from the underlying file.

11. A storage processor according to claim 10 , wherein assigning the storage objects and their respective files to version sets includes recording, in a version set database, the version set identifiers in association with file identifiers of files, the version set identifiers identifying the respective version sets, the file identifiers for each version set identifier identifying the files assigned to the respective version set, and wherein using the version set identifier to identify the files whose corresponding storage objects are assigned to the version set with which the block is associated includes applying the version set identifier for the block to the version set

Assignees

• Emc Corp

Inventors

• Bono Jean-Pierre
• Davenport William C
• Mathews Alexander
• Armangau Philippe

Classifications

• G06F3/065 Primary

  Replication mechanisms · CPC title

• G06F12/023 Primary

  Free address space management · CPC title

• G06F3/0655

  Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0619

  in relation to data integrity, e.g. data losses, bit errors · CPC title