Genomic application data storage

Patent metadata

Field	Value
Publication number	US-10324907-B2
Application number	US-201514747347-A
Country	US
Kind code	B2
Filing date	Jun 23, 2015
Priority date	Mar 14, 2013
Publication date	Jun 18, 2019
Grant date	Jun 18, 2019

Abstract

It is decided whether to increase a total amount of storage in a pool of Hadoop storage and whether to increase a total amount of processing in a pool of Hadoop processing. If it is decided to increase the total amount of storage and not increase the total amount of processing, the total amount of storage is increased without increasing processing. If it is decided to not increase the total amount of storage and increase the total amount of processing, the total amount of processing is increased without increasing storage. In response to receiving a request to perform a process on a set of data, processing is allocated from the pool of processing and storage is allocated from the pool of storage where the allocated processing and storage are used to perform the process on the set of data.

First claim

What is claimed is:

1. A system, comprising:
a hardware processor; and
a memory coupled with the hardware processor, wherein the memory is configured to provide the hardware processor with instructions which when executed cause the processor to:
decide whether to increase a total amount of storage in a pool of Hadoop storage;
decide whether to increase a total amount of processing in a pool of Hadoop processing;
in the event it is decided to: (1) increase the total amount of storage in the pool of Hadoop storage and (2) not increase the total amount of processing in the pool of Hadoop processing, increase the total amount of storage in the pool of Hadoop storage without increasing the total amount of processing in the pool of Hadoop processing; and
in the event it is decided to: (1) not increase the total amount of storage in the pool of Hadoop storage and (2) increase the total amount of processing in the pool of Hadoop processing, increase the total amount of processing in the pool of Hadoop processing without increasing the total amount of storage in the pool of Hadoop storage, wherein in response to receiving a request to perform a process on a set of data:
one or more storage resources from the pool of Hadoop storage is allocated, including a Hadoop file system; and
one or more processing resources from the pool of Hadoop processing is allocated, including a virtual and customized Hadoop compute node wherein an application or toolkit which runs on the virtual and customized Hadoop compute node is decoupled from a specific implementation of an underlying processor below a virtualization, including by:
storing a Hadoop compute node as a template;
deploying the Hadoop compute node stored as the template in order to obtain a default virtual Hadoop compute node; and
customizing the default virtual Hadoop compute node in order to obtain the virtual and customized Hadoop compute node, including by using a custom script to link the virtual and customized Hadoop compute node to the Hadoop file system.

2. The system recited in claim 1 , wherein the pool of Hadoop processing includes one or more of the following: virtual processing or Greenplum HD.

3. The system recited in claim 1 , wherein the pool of Hadoop storage includes one or more of the following: virtual storage, Isilon, a storage technology that supports a plurality of file system protocols on a single storage platform, or a storage technology that supports petabytes of storage.

4. The system recited in claim 1 , wherein the set of data includes genome data and a genome analysis toolkit runs on the allocated processing and the allocated storage.

5. A method, comprising:
using a processor to decide whether to increase a total amount of storage in a pool of Hadoop storage;
using the processor to decide whether to increase a total amount of processing in a pool of Hadoop processing;
in the event it is decided to: (1) increase the total amount of storage in the pool of Hadoop storage and (2) not increase the total amount of processing in the pool of Hadoop processing, using the processor to increase the total amount of storage in the pool of Hadoop storage without increasing the total amount of processing in the pool of Hadoop processing; and
in the event it is decided to: (1) not increase the total amount of storage in the pool of Hadoop storage and (2) increase the total amount of processing in the pool of Hadoop processing, using the processor to increase the total amount of processing in the pool of Hadoop processing without increasing the total amount of storage in the pool of Hadoop storage, wherein in response to receiving a request to perform a process on a set of data:
one or more storage resources from the pool of Hadoop storage is allocated, including a Hadoop file system; and
one or more processing resources from the pool of Hadoop processing is allocated, including a virtual and customized Hadoop compute node wherein an application or toolkit which runs on the virtual and customized Hadoop compute node is decoupled from a specific implementation of an underlying processor below a virtualization, including by:
storing a Hadoop compute node as a template;
deploying the Hadoop compute node stored as the template in order to obtain a default virtual Hadoop compute node; and
customizing the default virtual Hadoop compute node in order to obtain the virtual and customized Hadoop compute node, including by using a custom script to link the virtual and customized Hadoop compute node to the Hadoop file system.

6. The method recited in claim 5 , wherein the pool of Hadoop processing includes one or more of the following: virtual processing or Greenplum HD.

7. The method recited in claim 5 , wherein the pool of Hadoop storage includes one or more of the following: virtual storage, Isilon, a storage technology that supports a plurality of file system protocols on a single storage platform, or a storage technology that supports petabytes of storage.

8. The method recited in claim 5 , wherein the set of data includes genome data and a genome analysis toolkit runs on the allocated processing and the allocated storage.

9. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
deciding whether to increase a total amount of storage in a pool of Hadoop storage;
deciding whether to increase a total amount of processing in a pool of Hadoop processing;
in the event it is decided to: (1) increase the total amount of storage in the pool of Hadoop storage and (2) not increase the total amount of processing in the pool of Hadoop processing, increasing the total amount of storage in the pool of Hadoop storage without increasing the total amount of processing in the pool of Hadoop processing; and
in the event it is decided to: (1) not increase the total amount of storage in the pool of Hadoop storage and (2) increase the total amount of processing in the pool of Hadoop processing, increasing the total amount of processing in the pool of Hadoop processing without increasing the total amount of storage in the pool of Hadoop storage, wherein in response to receiving a request to perform a process on a set of data:
one or more storage resources from the pool of Hadoop storage is allocated, including a Hadoop file system; and
one or more processing resources from the pool of Hadoop processing is allocated, including a virtual and customized Hadoop compute node wherein an application or toolkit which runs on the virtual and customized Hadoop compute node is decoupled from a specific implementation of an underlying processor below a virtualization, including by:
storing a Hadoop compute node as a template;
deploying the Hadoop compute node stored as the template in order to obtain a default virtual Hadoop compute node; and
customizing the default virtual Hadoop compute node in order to obtain the virtual and customized Hadoop compute node, including by using a custom script to link the virtual and customized Hadoop compute node to the Hadoop file system.

10. The computer program product recited in claim 9 , wherein the pool of Hadoop processing includes one or more of the following: virtual processing or Greenplum HD.

11. The computer program product recited in claim 9 , wherein the pool of Hadoop storage includes one or more of the following: virtual storage, Isilon, a storage technology that supports a plurality of file system protocols on a single storage platform, or a storage technology that supports petabytes of storage.

12. The computer program product recited in claim 9 , wherein the set of data i

Assignees

• Emc Ip Holding Co Llc

Inventors

• Djam Hamid R
• Stewart Craig
• Hancock Michael
• Chen Qiyan
• Guo Feng
• Yan Kay
• Xiang Dong

Classifications

• G06F9/5072

  Grid computing · CPC title

• G16B50/00

  ICT programming tools or database systems specially adapted for bioinformatics · CPC title

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F9/50

  Allocation of resources, e.g. of the central processing unit [CPU] · CPC title

• G06F16/185 Primary

  Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof (details of archiving G06F16/11) · CPC title