File system block-level tiering and co-allocation

Patent metadata

Field	Value
Publication number	US-9952808-B2
Application number	US-201514669565-A
Country	US
Kind code	B2
Filing date	Mar 26, 2015
Priority date	Mar 26, 2015
Publication date	Apr 24, 2018
Grant date	Apr 24, 2018

Abstract

Embodiments of the invention relate to intra-block organized storage placement. One embodiment includes obtaining a file in a file system. The file is separated into multiple blocks. The multiple blocks are separated into at least two correlated sub-blocks. Intra-file block organized storage placement on different memory devices is determined for the at least two correlated sub-blocks in a file system metadata layout.

First claim

What is claimed is:

1. A method comprising:
obtaining a file in a file system;
separating the file into a plurality of blocks based on horizontally partitioning columns into multiple row groups;
separating each block of the plurality of blocks into at least two correlated sub-blocks, wherein one sub-block contains a subset of popular projection columns and another sub-block contains remaining columns; and
determining intra-file block organized storage placement on different types of memory devices for the at least two correlated sub-blocks in a file system metadata layout, wherein the intra-file block organized storage placement is based on a predictive column data block placement model that considers a plurality of attributes of column data blocks.

2. The method of claim 1 , wherein:
determining intra-file block organized storage placement comprises determining intra-file block tiering on the different types of memory devices for the at least two correlated sub-blocks;
the file system metadata comprises a block-level metadata header, a row-group-level metadata header and a column-level metadata header; and
the plurality of attributes of column data blocks comprises cardinality, sort order, sparsity and size.

3. The method of claim 2 , wherein:
determining intra-file block organized storage placement comprises determining intra-file block co-location on the different types of memory devices for the at least two correlated sub-blocks;
the block-level metadata header contains a number of row groups in a file block and a version number that aids in maintaining backward and forward compatibility across multiple versions of structure of the file block;
the row-group-level metadata header contains number of rows and columns present in a row group; and
the column-level metadata header maintains file offset pointers for multiple columnar layouts in the file block.

4. The method of claim 3 , further comprising:
maintaining a relationship between the at least two correlated sub-blocks, wherein one sub-block contains a subset of popular columns and a second sub-block contains remaining columns; and
co-placing the popular blocks and the unpopular blocks during file ingest, memory restriping, file migration, and file block up-tiering.

5. The method of claim 4 , further comprising:
providing an advisory by an application for placing a sub-block containing a subset of popular columns on a solid state drive (SSD) pool and a sub-block containing remaining columns on a hard disk drive (HDD) pool; and
specifying a constraint that the sub-block containing the subset of popular columns and the sub-block containing remaining columns for a specified number of rows, be placed on different storage pools on a same data node server in the file system.

6. The method of claim 2 , wherein a modified inode structure supports intra-file block tiering, and includes an entry that includes information indicating a co-location relationship between tiered sub-blocks residing on a same data node server in the file system.

7. The method of claim 2 , further comprising:
determining a first sub-block of the at least two correlated sub-blocks for the file targeted for a first memory tier;
determining a second sub-block of the at least two correlated sub-blocks for the file targeted for a second memory tier; and
selecting a data node server that has space for both the first sub-block on the first memory tier and the second sub-block on the second memory tier.

8. The method of claim 2 , wherein valid intra-file block tiering is differentiated from in-transition migration of the file across tiers or a failed migration.

9. The method of claim 1 , wherein determining intra-file block organized storage placement comprises determining intra-file block tiering and co-location on the different types of memory devices for the at least two correlated sub-blocks, and column data blocks are formatted in one of a first format optimized for fast selection for popular selection columns, a second format optimized for fast projection for popular projection columns, and a third format comprising a hybrid layout used for columns that are used as projection columns, selection columns, or both projection columns and selection columns.

10. The method of claim 1 , further comprising determining application advisory in the file system for determining block storage placement on the different types of memory devices for the at least two correlated sub-blocks.

11. The method of claim 10 , wherein an application controls sub-block placement for the at least two correlated sub-blocks on a desired tier at a time of file ingest for providing upfront performance guarantees.

12. A computer program product for intra-file block memory management, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by a processor to:
obtain, by the processor, a file in a file system;
separate, by the processor, the file into a plurality of blocks based on horizontally partitioning columns into multiple row groups;
separate, by the processor, each block of the plurality of blocks into at least two correlated sub-blocks, wherein one sub-block contains a subset of popular projection columns and another sub-block contains remaining columns; and
determine, by the processor, intra-file block organized storage placement on different types of memory devices for the at least two correlated sub-blocks in a file system metadata layout, wherein the intra-file block organized storage placement is based on a predictive column data block placement model that considers a plurality of attributes of column data blocks.

13. The computer program product of claim 12 , wherein:
determining intra-file block organized storage placement comprises determining intra-file block tiering on the different types of memory devices for the at least two correlated sub-blocks;
the file system metadata comprises a block-level metadata header, a row-group-level metadata header and a column-level metadata header; and
the plurality of attributes of column data blocks comprises cardinality, sort order, sparsity and size.

14. The computer program product of claim 13 , wherein:
determining intra-file block organized storage placement comprises determining intra-file block co-location on the different types of memory devices for the at least two correlated sub-blocks;
the block-level metadata header contains a number of row groups in a file block and a version number that aids in maintaining backward and forward compatibility across multiple versions of structure of the file block;
the row-group-level metadata header contains number of rows and columns present in a row group;
the column-level metadata header maintains file offset pointers for multiple columnar layouts in the file block; and
column data blocks are formatted in one of a first format optimized for fast selection for popular selection columns, a second format optimized for fast projection for popular projection columns, and a third format comprising a hybrid layout used for columns that are used as projection columns, selection columns, or both projection columns and selection columns.

15. The computer program product of claim 14 , wherein the program code executable by the processor to further:
maintain a relationship between a first sub-block of the at least two correlated sub-blocks that contains a subset of popular columns and a second sub-block of the at least two correlated sub-blocks that contains remaining columns;
co-place and manage together, by the pr

Assignees

• IBM

Inventors

• Kaushik Rini

Classifications

• G06F3/061

  Improving I/O performance · CPC title

• G06F3/0643

  Management of files · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F16/113

  Details of archiving (lifecycle management in storage systems G06F3/0649; point-in-time backing up or restoration of persistent data G06F11/1446) · CPC title

• G06F3/0685 Primary

  Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays · CPC title