Tiering with pluggable storage system for parallel query engines

Patent metadata

Field	Value
Publication number	US-11514046-B2
Application number	US-202016865164-A
Country	US
Kind code	B2
Filing date	May 1, 2020
Priority date	Feb 25, 2013
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes determining a usage level of a file, wherein the file is stored in a first storage system, moving the file to a second storage system based on the determined usage level of the file updating location information in a catalog based on the movement of the file, and performing at least a portion of a query on the file after updating location information in the catalog.

First claim

What is claimed is:

1. A computer implemented method, comprising:
receiving, by a universal node comprising one or more processors, a query from a client;
accessing, by the universal node, a catalog comprising location information for one or more files stored across one or more storage systems, the one or more storage systems comprising a first storage system and a second storage system, and the universal node being configured to interface with the first storage system and the second storage system; and
providing, by the universal node, a universal namespace to the client, wherein the universal namespace collectively presents information for the one or more files stored in a plurality of namespaces corresponding to the one or more storage systems, and a file of the one or more files is accessible via an address with respect to the universal namespace, and the catalog stores a mapping of addresses for files with respect to the universal namespace to the corresponding location information on the one or more storage systems.

2. The method of claim 1 , wherein the file of the one or more files is accessible via the universal namespace such that the client accesses the file at a same address of the universal namespace regardless of whether the file is moved from the first storage system to the second storage system.

3. The method of claim 1 , wherein a catalog service updates the catalog with a location of the file based on a storage system of the plurality of storage systems on which the file is stored.

4. The method of claim 1 , further comprising:
determining, by one or more processors, a usage level of the file, wherein the file is stored in the first storage system; and
moving, by one or more processors, the file to the second storage system based at least in part on the determined usage level of the file.

5. The method of claim 4 , further comprising:
updating, by one or more processors, the mapping of the address for the file to location information on the one or more storage systems based on the movement of the file.

6. The method of claim 4 , further comprising:
performing at least a portion of the query on the file after the mapping of the address for the file to location information on the one or more storage systems is updated, wherein the universal node comprises a universal job tracker that tracks a status of a one or more jobs corresponding to the query.

7. The method of claim 6 , wherein the universal job tracker obtains a corresponding status of at least a portion of the query from one or more data node job trackers.

8. The method of claim 4 , wherein moving the file to the second storage system based on the determined usage level includes moving the file to a high performance storage system based on high usage level.

9. The method of claim 8 , wherein the second storage system is a higher performance storage system than the first storage system.

10. The method of claim 4 , wherein the usage level includes a usage frequency.

11. The method of claim 4 , further comprising determining a predicted usage time based on the usage frequency.

12. The method of claim 11 , wherein the moving the file to the second storage system based on the determined usage level includes moving the file to the second storage system based at least in part on the predicted usage time.

13. The method of claim 1 , wherein the universal node is configured to interface with the first storage system and the second storage system based at least in part on a universal protocol plugin.

14. The method of claim 1 , wherein to access the file the client does not directly use an address of the file in relation to the particular storage system of the plurality of storage system on which the file is located.

15. The method of claim 1 , wherein the universal node comprises a plurality of protocols for a plurality of file systems, and the universal node uses corresponding ones of the plurality of protocols in connection with accessing the first storage system and the second storage system.

16. The method of claim 1 , further comprising:
in response to receiving the query, determining a location of the one or more files in the universal namespace using the catalog; and
in response to a determination that the one or more files are located at the first storage system, sending a job corresponding to at least a portion of the query to the first storage system.

17. A system, comprising:
one or more processors configured to:
receive a query from a client;
access a catalog comprising location information for one or more files stored across one or more storage systems, the one or more storage systems comprising a first storage system and a second storage system, and a universal node being configured to interface with the first storage system and the second storage system; and
provide a universal namespace to the client, wherein the universal namespace collectively presents information for the one or more files stored in a plurality of namespaces corresponding to the one or more storage systems, and a file of the one or more files is accessible via an address with respect to the universal namespace, and the catalog stores a mapping of addresses for files with respect to the universal namespace to the corresponding location information on the one or more storage systems; and
a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.

18. The system of claim 17 , wherein the one or more processors are further configured to:
determine a usage level of the file, wherein the file is stored in the first storage system; and
move the file to the second storage system based at least in part on the determined usage level of the file.

19. The system of claim 18 , wherein the one or more processors are further configured to:
update the mapping of the address for the file to location information on the one or more storage systems based on the movement of the file.

20. A computer program product, comprising a non-transitory computer readable medium having program instructions implemented by a processor circuitry when executed, the program instructions comprising instructions for:
receiving, by a universal node comprising one or more processors, a query from a client;
accessing, by the universal node, a catalog comprising location information for one or more files stored across one or more storage systems, the one or more storage systems comprising a first storage system and a second storage system, and the universal node being configured to interface with the first storage system and the second storage system; and
providing, by the universal node, a universal namespace to the client, wherein the universal namespace collectively presents information for the one or more files stored in a plurality of namespaces corresponding to the one or more storage systems, and a file of the one or more files is accessible via an address with respect to the universal namespace, and the catalog stores a mapping of addresses for files with respect to the universal namespace to the corresponding location information on the one or more storage systems.

Assignees

• Emc Ip Holding Co Llc

Inventors

• Tiwari Sameer
• Bhandarkar Milind Arun
• Mogal Bhooshan Deepak

Classifications

• G06F16/245

  Query processing · CPC title

• G06F16/2455

  Query execution · CPC title

• G06F16/10

  File systems; File servers · CPC title

• G06F16/217

  Database tuning (G06F16/2282 takes precedence; database performance monitoring G06F11/3409) · CPC title

• G06F16/182

  Distributed file systems · CPC title