Adaptable data caching mechanism for in-memory cluster computing

Patent metadata

Field	Value
Publication number	US-2016170882-A1
Application number	US-201514712895-A
Country	US
Kind code	A1
Filing date	May 14, 2015
Priority date	Dec 16, 2014
Publication date	Jun 16, 2016
Grant date	—

Abstract

An in-memory cluster computing framework node is described. The node includes storage devices having various priorities. The node also includes a resource monitor to monitor the operation of the storage devices. The node also includes a resource scheduler. When the resource monitor indicates that a storage device is at or approaching saturation, the resource scheduler can migrate data from that storage device to another storage device of lower priority.

First claim

What is claimed is:

1 . An in-memory cluster computing framework node ( 305 ), comprising:
a processor ( 315 ); a first storage device ( 220 , 225 , 230 , 235 , 130 ) storing cached data, the first storage device ( 220 , 225 , 230 , 235 , 130 ) having a first priority ( 240 , 245 , 250 , 255 , 260 ); a second storage device ( 220 , 225 , 230 , 235 , 130 ) having a second priority ( 240 , 245 , 250 , 255 , 260 ); a resource monitor ( 205 ) operative to monitor the first storage device ( 220 , 225 , 230 , 235 , 130 ); and a resource scheduler ( 210 ) operative to migrate the cached data from the first storage device ( 220 , 225 , 230 , 235 , 130 ) to the second storage device ( 220 , 225 , 230 , 235 , 130 ) if the resource monitor ( 205 ) indicates that the first storage device ( 220 , 225 , 230 , 235 , 130 ) is saturated.

2 . An in-memory cluster computing framework node ( 305 ) according to claim 1 , wherein the resource monitor ( 205 ) is operative to determine the capabilities of the first storage device ( 220 , 225 , 230 , 235 , 130 ).

3 . An in-memory cluster computing framework node ( 305 ) according to claim 1 , wherein the resource scheduler ( 210 ) is operative to select the first storage device ( 220 , 225 , 230 , 235 , 130 ) to initially cache the data based on information provided by an application that uses the data.

4 . An in-memory cluster computing framework node ( 305 ) according to claim 1 , wherein:
the first priority ( 240 , 245 , 250 , 255 , 260 ) is higher than the second priority ( 240 , 245 , 250 , 255 , 260 ); and the resource scheduler ( 210 ) is operative to select the first storage device ( 220 , 225 , 230 , 235 , 130 ) to initially cache the data as a higher priority ( 240 , 245 , 250 , 255 , 260 ) device.

5 . An in-memory cluster computing framework node ( 305 ) according to claim 4 , wherein the resource scheduler is operative to select the second storage device ( 220 , 225 , 230 , 235 , 130 ) for future data caching if the resource monitor ( 205 ) indicates that the first storage device ( 220 , 225 , 230 , 235 , 130 ) is saturated.

6 . An in-memory cluster computing framework node ( 305 ) according to claim 1 , further comprising a replicator ( 310 ) to replicate the cached data on a third storage device ( 220 , 225 , 230 , 235 , 130 ).

7 . An in-memory cluster computing framework node ( 305 ) according to claim 6 , wherein the third storage device ( 220 , 225 , 230 , 235 , 130 ) is in a second in-memory cluster computing framework node ( 305 ).

8 . An in-memory cluster computing framework node ( 305 ) according to claim 1 , wherein the data includes a resilient distributed dataset (RDD) on the first storage device ( 220 , 225 , 230 , 235 , 130 ).

9 . An in-memory cluster computing framework node ( 305 ) according to claim 1 , wherein the resource scheduler ( 210 ) is operative to migrate all data from the first storage device ( 220 , 225 , 230 , 235 , 130 ) to the second storage device ( 220 , 225 , 230 , 235 , 130 ).

10 . An in-memory cluster computing framework node ( 305 ) according to claim 1 , wherein the resource scheduler ( 210 ) is operative to migrate an oldest data from the first storage device ( 220 , 225 , 230 , 235 , 130 ) to the second storage device ( 220 , 225 , 230 , 235 , 130 ).

11 . A method for caching data in an in-memory cluster computing framework, comprising:
caching ( 510 ) a data on a first storage device ( 220 , 225 , 230 , 235 , 130 ) with a first priority ( 240 , 245 , 250 , 255 , 260 ) in a cluster node ( 305 ); monitoring ( 520 ) the operation of the first storage device ( 220 , 225 , 230 , 235 , 130 ); and if the first storage device ( 220 , 225 , 230 , 235 , 130 ) is saturated, migrating ( 530 ) the cached data to a second storage device ( 220 , 225 , 230 , 235 , 130 ) with a second priority ( 240 , 245 , 250 , 255 , 260 )).

12 . A method according to claim 11 , wherein monitoring ( 520 ) the operation of the first storage device ( 220 , 225 , 230 , 235 , 130 ) includes determining ( 505 ) a capability of the first storage device ( 220 , 225 , 230 , 235 , 130 ).

13 . A method according to claim 11 , wherein caching ( 510 ) a data on a first storage device ( 220 , 225 , 230 , 235 , 130 ) with a first priority ( 240 , 245 , 250 , 255 , 260 ) in a cluster node ( 305 ) includes caching ( 510 ) the data on the first storage device ( 220 , 225 , 230 , 235 , 130 ) in the cluster node ( 305 ), the first storage device ( 220 , 225 , 230 , 235 , 130 ) selected by an application using the data.

14 . A method according to claim 11 , wherein caching ( 510 ) a data on a first storage device ( 220 , 225 , 230 , 235 , 130 ) with a first priority ( 240 , 245 , 250 , 255 , 260 ) in a cluster node ( 305 ) includes caching ( 510 ) the data on the first storage device ( 220 , 225 , 230 , 235 , 130 ) in the cluster node ( 305 ), the first storage device ( 220 , 225 , 230 , 235 , 130 ) having a higher priority ( 240 , 245 , 250 , 255 , 260 ) among a plurality of devices.

15 . A method according to claim 14 , further comprising, if the first storage device ( 220 , 225 , 230 , 235 , 130 ) is saturated, re-directing ( 535 ) future cache requests for the first storage device ( 220 , 225 , 230 , 235 , 130 ) in the cluster node ( 305 ) in the cluster node ( 305 ) to the second storage device ( 220 , 225 , 230 , 235 , 130 ).

16 . A method according to claim 11 , further comprising replicating ( 515 ) the cached data on a third storage device ( 220 , 225 , 230 , 235 , 130 ).

17 . A method according to claim 16 , wherein replicating ( 515 ) the cached data on a third storage device ( 220 , 225 , 230 , 235 , 130 ) includes replicating ( 515 ) the cached data on the third storage device ( 220 , 225 , 230 , 235 , 130 ) in a second cluster node ( 305 ).

18 . A method according to claim 11 , wherein caching ( 510 ) a data on a first storage device ( 220 , 225 , 230 , 235 , 130 ) with a first priority ( 240 , 245 , 250 , 255 , 260 ) in a cluster node ( 305 ) includes caching a resilient distributed dataset (RDD) on the first storage device ( 220 , 225 , 230 , 235 , 130 ).

19 . A method according to claim 11 , wherein migrating ( 530 ) the cached data to a second storage device ( 220 , 225 , 230 , 235 , 130 ) with a second priority ( 240 , 245 , 250 , 255 , 260 ) includes migrating ( 705 ) all data on the first storage device ( 220 , 225 , 230 , 235 , 130 ) to the second storage device ( 220 , 225 , 230 , 235 , 130 ).

20 . A method according to claim 11 , wherein migrating ( 530 ) the cached data to a second storage device ( 220 , 225 , 230 , 235 , 130 ) with a second priority ( 240 , 245 , 250 , 255 , 260 ) includes migrating ( 710 ) an oldest data on the first storage device ( 220 , 225 , 230 , 235 , 130 ) to the second storage device ( 220 , 225 , 230 , 235 , 130 ).

Assignees

• Samsung Electronics Co Ltd

Inventors

• Choi Inseok Stephen
• Ki Yang Seok

Classifications

• G06F3/0647

  Migration mechanisms · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F11/3034

  where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097) · CPC title

• G06F3/0608

  Saving storage space on storage systems · CPC title

• G06F3/0653

  Monitoring storage devices or systems · CPC title