Parallel data processing system, computer, and parallel data processing method

Patent metadata

Field	Value
Publication number	US-9841989-B2
Application number	US-201214404550-A
Country	US
Kind code	B2
Filing date	May 31, 2012
Priority date	May 31, 2012
Publication date	Dec 12, 2017
Grant date	Dec 12, 2017

Abstract

A parallel data processing system includes a parallel data processing execution unit for reading a data from a data set including a first data set that includes a plurality of first data and a second data set that includes a plurality of second data and executing processing. The parallel data processing execution unit (A) reads the first data from the first data set, and acquires a first value from the first data based on first format information acquired from an application, (B) generates one or more threads for respectively reading one or more second data corresponding to the first value from the second data set based on first reference information acquired from the application, (C) executes (A) and (B) on one or more first data in the first data set, and (D) executes a plurality of the threads in parallel.

First claim

The invention claimed is:

1. A parallel data processing system to execute parallel data processing, the parallel data processing system comprising:
a plurality of computers connected over a network, each of the computers including a processor connected to a memory; and
one or more data storage media connected to the network and which store a first data set that includes a plurality of first data, a second data set that includes a plurality of second data, and a third data set that includes a plurality of third data, where at least one of the first data corresponds to one or more of the second data and includes first reference information that represents a reference to the one or more of the second data of the second data set corresponding to the first data, and at least one of the second data corresponds to one or more of the third data and includes second reference information that represents a reference to the one or more third data of the third data set corresponding to the second data,
wherein the memory of at least one of the computers stores instructions that, when executed by the processor connected therewith, cause the processor connected therewith to:
execute an application which has first format information, second format information, third format information, a first reference scheme, and a second reference scheme, where the first format information, the second format information and the third format information represent respective methods for interpreting data in the first data set, the second data set, and the third data set, and the application causes a process to read the first data set, the second data set, and the third data set,
wherein the first reference scheme represents a respective method for acquiring data in the second data set in accordance with the first reference information in the first data set and the second reference scheme represents a respective method for acquiring data in the third data set in accordance with the second reference information in the second data set,
wherein the memory of the at least one of the computers stores instructions that, when executed by the processor connected therewith, cause the processor connected therewith to:
(A) read the first data from the first data set, and acquire a first value from the first data based on the first format information acquired from the application,
(B) determine whether or not the first value includes the first reference information,
(C) specify, when the first value includes the first reference information, one or more second data corresponding to the first data based on the first reference information,
(D) generate one or more threads for the specified one or more of the second data, and read the second data based on the first reference scheme acquired from the application,
(E) acquire a second value from the second data based on the second format information acquired from the application,
(F) determine whether or not the second value includes the second reference information,
(G) specify, when the second value includes the second reference information, one or more of the third data corresponding to the second data based on the second reference information,
(H) generate one or more threads for the specified one or more of the third data, and read the third data based on the second reference scheme acquired from the application,
(I) acquire a third value from the third data based on the third format information acquired from the application,
(J) execute (A), (B), (C), (D), (E), (F), (G), (H) and (I) on one or more of the first data in the first data set so that a plurality of threads corresponding to the one or more of the first data are generated, and
(K) execute the plurality of the threads corresponding to the one or more first data in parallel,
wherein the first format information is a program code, and
wherein the memory of at least one of the computers further stores instructions that, when executed by the processor connected therewith, cause the processor connected therewith to:
receive catalogue information required for generating the first format information written in a predetermined markup language from a user, and
generate the first format information based on the catalogue information.

2. The parallel data processing system according to claim 1 ,
wherein output data is generated from the second value acquired from one or more of the second data.

3. The parallel data processing system according to claim 1 ,
wherein the first reference information includes information for identifying a physical position at which the second data is stored in the second data set.

4. The parallel data processing system according to claim 1 ,
wherein the first reference information includes information for searching the second data in the second data set.

5. The parallel data processing system according to claim 1 ,
wherein at least part of the second data set is distributed among the storage media, and
wherein the second data is acquired from the storage media before acquiring the second value in (E).

6. The parallel data processing system according to claim 1 , wherein the memory of the at least one of the computers stores instructions that, when executed by the processor connected therewith, cause the processor connected therewith to:
issue an instruction of processing from the application which defines a procedure, and
execute the processing of (A) to (K) upon receiving the instruction, so as to execute the processing in an order different from that specified in the procedure.

7. A computer in a computer system in which a plurality of computers are connected by a network and the computer system stores a first data set that includes a plurality of first data, a second data set that includes a plurality of second data, and a third data set that includes a plurality of third data, where at least one of the first data corresponds to one or more of the second data and includes first reference information that represents a reference to the one or more of the second data of the second data set corresponding to the first data, and at least one of the second data corresponds to one or more of the third data and includes second reference information that represents a reference to the one or more third data of the third data set corresponding to the second data,
where the computer system executes an application which has first format information, second format information, third format information, a first reference scheme, and a second reference scheme,
where the first format information, the second format information and the third format information represent respective methods for interpreting data in the first data set, the second data set, and the third data set, and the application causes a process to read the first data set, the second data set, and the third data set,
where the first reference scheme represents a respective method for acquiring data in the second data set in accordance with the first reference information in the first data set and the second reference scheme represents a respective method for acquiring data in the third data set in accordance with the second reference information in the second data set,
where the application specifying a process to be executed on the first data set, the second data set, and the third data set, the computer comprising:
a communication interface device connected to the computer system over the network;
a processor that is connected to the communication interface; and
a memory that stores instructions that, when executed by the processor, cause the processor to:
(A) read the first data from the first data set, and acquire a first value from the first data based on the first format information acquired from t

Assignees

• Goda Kazuo
• Yamada Hiroyuki
• Kitsuregawa Masaru
• Kawamura Nobuo
• Fujiwara Shinji
• Mogi Kazuhiko
• Hitachi Ltd
• Univ Tokyo

Inventors

• Goda Kazuo
• Yamada Hiroyuki
• Kitsuregawa Masaru
• Kawamura Nobuo
• Fujiwara Shinji
• Mogi Kazuhiko

Classifications

• G06F9/466 Primary

  Transaction processing · CPC title

• G06F9/4843 Primary

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F17/30445

  Physics · mapped topic

• G06F9/54

  Interprogram communication · CPC title

• G06F17/30477

  Physics · mapped topic