System and method for managing data in dispersed systems

Patent metadata

Field	Value
Publication number	US-10534757-B2
Application number	US-201615205673-A
Country	US
Kind code	B2
Filing date	Jul 8, 2016
Priority date	Mar 2, 2016
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

Provided is a data management system including a master group including a data manager configured to manage data, a process manager configured to manage a process of analyzing the data, and a meta data storage unit configured to store meta data information for managing the data; and at least one work group including a work manager configured to manage resources, the data, and a data process on the basis of a scope of groups, and at least one node manager configured to manage the resources, the data, and the data process on the basis of a scope of nodes. The master group and the at least one work group are dispersed from each other.

First claim

What is claimed is:

1. A data management system comprising:
a master group including a processor, the master group configured to:
receive, from a user, a first request indicating an operation and including a data identifier, the operation being one of a read operation, a write operation, and a process operation, and when the first request is valid:
produce a list of appropriate work groups, the list of appropriate work groups including 1) one or more identifiers of work groups appropriate for performing the operation on the data identified by the data identifier and 2) one or more identifiers of nodes included in the identified work groups, and provide the list of appropriate work groups to the user; and
at least one work group including a processor and a plurality of nodes, wherein the at least one work group identified in the list of appropriated work groups, the work group configure to: and
receive, from the user, a second request specifying the operation and including a list information of nodes,
determine, using the list information of nodes, a list of information regarding valid nodes for performing the operation,
provide the list of information regarding valid nodes to the user,
receive, from the user, information of at least one node selected from the valid nodes, and
perform the operation using the selected at least one node, wherein the master group and the at least one work group are dispersed from each other.

2. The data management system of claim 1 ,
wherein the master group is further configured to determine whether the first request is valid using meta data information stored in the master group, and
wherein the meta data information comprises at least one of:
user information;
information regarding the at least one work group;
information regarding an amount of works allocated to the at least one work group;
information regarding the resources;
information regarding an application; and
configuration information of clusters.

3. The data management system of claim 1 , wherein the master group and the at least one work group are activated by being connected to each other when a request is received from a user.

4. The data management system of claim 1 , wherein when the operation is the write operation:
the list of appropriate work groups includes one or more identifiers of work groups appropriate for writing data indicated by the data identifier and arranged in a rank order, and
the list of appropriate work groups is produced based on at least one of:
location information of the user;
information regarding use of the resources; and
information regarding an amount of works allocated to the at least one work group.

5. The data management system of claim 4 , wherein when the operation is the write operation:
the at least one work group is further configured to receive, from the user, data corresponding to the data identifier, and
performing the operation using the selected at least one node includes writing the received data to the selected at least one node.

6. The data management system of claim 5 , wherein when the operation is the write operation, performing the operation using the selected at least one node includes copying the data and transmitting the copied data to another node or to another work group.

7. The data management system of claim 6 , wherein when the operation is the write operation, the copied data is transmitted to a node or work group selected randomly, selected based on a use pattern, or selected by the user.

8. The data management system of claim 6 , wherein when the operation is the write operation, the at least one work group transmits location information of the copied data to the master group, and
wherein the master group is configured to update, using the location information, meta data information stored in the master group.

9. The data management system of claim 1 , wherein when the operation is the read operation:
the list of appropriate work groups includes one or more identifiers of work groups appropriate for reading data identified by data identifier and arranged in a rank order, and
the list of appropriate work groups is produced based on at least one of:
location information of the user;
information regarding use of the resources; and
information regarding an amount of works allocated to the at least one work group.

10. The data management system of claim 9 , wherein when the operation is the read operation, performing the operation using the selected at least one node includes transmitting the data identified by the data identifier to the user.

11. The data management system of claim 1 , wherein when the operation is the process operation:
the first request includes a process identifier;
the second request includes the process identifier,
the data identifier includes an input data identifier and an output data identifier;
the list of appropriate work groups includes one or more identifiers of work groups appropriate to handle a process corresponding to the process identifier and arranged in a rank order; and
the list of appropriate work groups is produced based on at least one of:
location information of the user;
information regarding use of the resources; and
information regarding an amount of works allocated to the work groups.

12. The data management system of claim 11 , wherein when the operation is the process operation, perform the operation includes transmitting a data process allocation request to a node manager corresponding to the selected at least one node,
wherein the node manager includes a processor and is configured to process data identified by the data identifier in response to receiving the data process allocation request.

13. The data management system of claim 1 , wherein the master group is further configured to authenticate the user in response to an authentication request from the user, and
wherein receive, from the user, the first request is performed after authenticating the user.

14. A data management method performed by a data management system, comprising:
receiving, by a master group from a user, a first request from a user, the first request including an indication of data and indicating an operation, the operation being one of a read operation, a write operation, and a process operation;
producing a list of work groups corresponding to the first request, and arranged in a rank order, the list of work groups including one or more identifiers of work groups and a list of nodes included in the identified work groups;
providing, to the user by the master group, the list of work groups;
receiving, from the user by at least one work group identified in the list of work groups, a second request indicating the operation and including list information of nodes;
when the operation is the write operation or the read operation, transmitting, by the at least one work group, information regarding valid nodes to the user in response to the second request, based on the list information of nodes; and
when the operation is the process operation, processing, by the at least one work group, the data according to the list information of nodes,
wherein the master group and the at least one work group are dispersed from each other.

15. The data management method of claim 14 , wherein when the operation is the write operation, the list information of nodes indicates on or more nodes to which the data is to be written,
wherein when the operation is the read operation, the list information of nodes indicates on or more nodes from which the data is to be read, and
wherein when the oper

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Chau Nguyen Minh
• Won Hee Sun

Classifications

• H04L63/08 Primary

  for authentication of entities (cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32) · CPC title

• G06F16/24578

  using ranking · CPC title

• G06F16/182 Primary

  Distributed file systems · CPC title

• G06F16/285 Primary

  Clustering or classification · CPC title