Tiering with pluggable storage system for parallel query engines

Patent metadata

Field	Value
Publication number	US-9898475-B1
Application number	US-201313842955-A
Country	US
Kind code	B1
Filing date	Mar 15, 2013
Priority date	Feb 25, 2013
Publication date	Feb 20, 2018
Grant date	Feb 20, 2018

Abstract

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes determining a usage level of a file, wherein the file is stored in a first storage system, moving the file to a second storage system based on the determined usage level of the file updating location information in a catalog based on the movement of the file, and performing at least a portion of a query on the file after updating location information in the catalog.

First claim

What is claimed is:

1. A computer implemented method for managing data, comprising:
determining, by a processor circuitry, a usage level of a file, wherein the file is stored in a first storage system;
moving, by the processor circuitry, the file to a second storage system based on the determined usage level of the file;
updating, by a catalog service, location information in a catalog based on the movement of the file, wherein the catalog stores location information indicating a location for a plurality of files located at a plurality storage systems corresponding to a plurality of namespaces, wherein the catalog is accessed by a universal node that is configured to interface with the first storage system and the second storage system, and wherein the universal node provides, to a client, a universal namespace across the first storage system and the second storage system so as to collectively present the plurality of namespaces as the universal namespace, wherein the file is accessible via the universal namespace such that the client accesses the file at a same address of the universal namespace regardless of whether the file is moved from the first storage system to the second storage system; and
performing at least a portion of a query on the file after updating location information in the catalog, wherein the universal node comprises a universal job tracker that tracks a status of a one or more jobs corresponding to the query.

2. The method as recited in claim 1 , wherein moving the file to the second storage system based on the determined usage level includes moving the file to a high performance storage system based on high usage level.

3. The method as recited in claim 2 , wherein the second storage system is a higher performance storage system than the first storage system.

4. The method as recited in claim 1 , wherein the usage level includes a usage frequency.

5. The method as recited in claim 4 , further comprising determining a predicted usage time based on the usage frequency.

6. The method as recited in claim 5 , wherein moving the file to the second storage system based on the determined usage level includes moving the file to the second storage system before the predicted usage time.

7. The method as recited in claim 6 , further comprising moving the file to the first storage system after the predicted usage time.

8. The method of claim 1 , wherein the first storage system and the second storage system use different file systems.

9. The method of claim 1 , wherein in the event that the file is moved from the first storage system to the second storage system, the file is accessible via the universal namespace and the move of the file from the first storage system to the second storage system is transparent to the client.

10. The method of claim 8 , wherein the universal node comprises a plurality of protocols for a plurality of file systems, and wherein the universal node uses corresponding ones of the plurality of protocols in connection with accessing the first storage system and the second storage system.

11. The method of claim 1 , wherein the universal job tracker obtains a corresponding status of at least a portion of the query from one or more data node job trackers.

12. The method of claim 1 , further comprising:
receiving, by the universal node, the query from the client;
in response to receiving the query, determining a location of one or more files in the universal namespace using the catalog service; and
in the event that the one or more files are located at the first storage system, sending a job corresponding to at least a portion of the query to the first storage system.

13. The method of claim 1 , wherein in event that the client is to access the file, the client accesses the file via the universal namespace using the corresponding address of the file in relation to the universal namespace.

14. The method of claim 13 , wherein the client does not directly use an address of the file in relation to the particular storage system of the plurality of storage system on which the file is located.

15. A system for managing data, comprising a processor circuitry configured to:
determine a usage level of a file, wherein the file is stored in a first storage system;
move the file to a second storage system based on the determined usage level of the file;
update, by a catalog service, location information in a catalog based on the movement of the file, wherein the catalog stores location information indicating a location for a plurality of files located at a plurality storage systems corresponding to a plurality of namespaces, wherein the catalog is accessed by a universal node that is configured to interface with the first storage system and the second storage system and wherein the universal node provides, to a client, a universal namespace across the first storage system and the second storage system so as to collectively present the plurality of namespaces as the universal namespace, wherein the file is accessible via the universal namespace such that the client accesses the file at a same address of the universal namespace regardless of whether the file is moved from the first storage system to the second storage system; and
perform at least a portion of a query on the file after updating location information in the catalog, wherein the universal node comprises a universal job tracker that tracks a status of a one or more jobs corresponding to the query.

16. The system as recited in claim 15 , wherein move the file to the second storage system based on the determined usage level includes move the file to a high performance storage system based on high usage level.

17. The system as recited in claim 16 , wherein the second storage system is a higher performance storage system than the first storage system.

18. The system as recited in claim 15 , wherein the usage level includes a usage frequency.

19. The system as recited in claim 18 , the processor further configured to determine a predicted usage time based on the usage frequency.

20. The system as recited in claim 19 , wherein move the file to the second storage system based on the determined usage level includes move the file to the second storage system before the predicted usage time.

21. The system as recited in claim 20 , the processor further configured to move the file to the first storage system after the predicted usage time.

22. A computer program product for processing data, comprising a non-transitory computer readable medium having program instructions implemented by a processor circuitry when executed, the program instructions comprising instructions for:
determining a usage level of a file, wherein the file is stored in a first storage system;
moving the file to a second storage system based on the determined usage level of the file;
updating, by a catalog service, location information in a catalog based on the movement of the file, wherein the catalog stores location information indicating a location for a plurality of files located at a plurality storage systems corresponding to a plurality of namespaces, wherein the catalog is accessed by a universal node that is configured to interface with the first storage system and the second storage system and wherein the universal node provides, to a client, a universal namespace across the first storage system and the second storage system so as to collectively present the plurality of namespaces as the universal namespace, wherein the file

Assignees

• Emc Ip Holding Co Llc

Inventors

• Tiwari Sameer
• Bhandarkar Milind Arun
• Mogal Bhooshan Deepak

Classifications

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• G06F16/217

  Database tuning (G06F16/2282 takes precedence; database performance monitoring G06F11/3409) · CPC title

• G06F16/1858

  Parallel file systems, i.e. file systems supporting multiple processors · CPC title

• G06F16/24532

  of parallel queries · CPC title

• G06F16/10

  File systems; File servers · CPC title