Stream-processing data

Patent metadata

Field	Value
Publication number	US-10146502-B2
Application number	US-201514799608-A
Country	US
Kind code	B2
Filing date	Jul 15, 2015
Priority date	Sep 29, 2014
Publication date	Dec 4, 2018
Grant date	Dec 4, 2018

Abstract

A method for stream-processing data including a missing part in real time and thereafter updating the result of the stream processing. A technique for processing data is included. The technique includes receiving data; detecting a probably missing part in the received data while stream-processing the received data in real time; and comparing master data corresponding to the received data and having no missing part with the probably missing part, and if the received data has the missing part, updating the result of the stream processing using the master data.

First claim

What is claimed is:

1. A method for processing data, the method comprising:
receiving, by a processor of a computing system, data;
detecting, by the processor of the computing system, a probably missing part in the received data while stream-processing of the received data in real time by:
stream processing of the received data in a first stream processing line that is a combination of a plurality of operators, each operator of the plurality of operators being one processing unit;

comparing, by the processor of the computing system, master data corresponding to the received data and having no missing part with the probably missing part;
relocating, by the processor of the computing system, the combination of the plurality of operators to define a second stream processing line,
wherein the relocation of the combination of the plurality of operators is determined in accordance with at least one of a group consisting of a probability of the missing, a type of the missing, a range of influence of a result of the stream processing due to the missing, an order of priority of an output the result of the stream processing, a dependency relationship among operators of the combination of the plurality of operators, a processing time allowable for updating the result of the stream processing, or a system resource available for the updating the result of the stream processing; if the received data has the missing part, updating, by the processor of the computing system, the result of the stream processing by using the master data and the second stream processing line;

the method further comprising:
calculating an amount of the system resource required for execution of the combination of the plurality of relocated operators to process the received data in real time;
determining that the calculated required amount of the system resource exceeds a system resource availability threshold;
determining that the calculated required amount of the system resource is reducible to below the threshold;
reducing the calculated required amount of the system resource by deleting a portion of modules of a particular operator of the combination of the plurality of relocated operators in response to determining that the calculated required amount of the system resource is reducible to below the threshold; and
deploying the combination of the plurality of relocated operators according to the defined second stream processing line.

2. The method according to claim 1 , wherein the step of updating the result of the stream processing includes:
processing, by the processor of the computing system, the master data in accordance with the second stream processing line; and
using a result of the processing the master data to update the result of the stream processing.

3. The method according to claim 1 , the method further comprising:
locating, by the processor of the computing system, the relocated combination of the plurality of operators on one of a group consisting of a plurality of processor nodes on a computer system and a plurality of virtual processor nodes on the computer system.

4. The method according to claim 1 , wherein if at least one of a group consisting of a processing time allowable for the updating result is limited, or a system resource available for the updating the result is limited, the relocating of the combination of the plurality operators is determined in accordance with an order of priority of an output of the result.

5. The method according to claim 1 , wherein the detecting a probably missing part in the received data further comprises:
outputting, by the processor of the computing system, at least one of a group consisting of a range of data that may have a missing part, a probability of the missing, a type of the missing, a range of influence of the result due to the missing, an order of priority of an output of the result, a dependency relationship among operators of the combination of the plurality of operators, a processing time required for the operators to perform the stream processing, or a system resource required for the operators to perform the stream processing.

6. The method according to claim 1 , wherein the data is measured values from at least one sensor, and wherein the detecting the probably missing part in the received data includes:
detecting, by the processor of the computing system, the probably missing part using the measured values.

7. The method according to claim 1 , wherein the data is measured values from at least one sensor, and wherein the step of detecting the probably missing part in the received data further includes:
detecting, by the processor of the computing system, a collection interval of the measured values.

8. The method according to claim 7 , wherein the step of detecting the probably missing part in the received data further includes:
if a difference in the collection interval is detected, detecting data of a part in which the difference is detected as the probably missing part.

9. The method according to claim 8 , wherein the step of updating the result of the stream processing includes:
updating, by the processor of the computing system, within the result of the stream processing, a second result caused by the missing due to the difference in the collection interval, using a third result obtained from the master data.

10. The method according to claim 1 , wherein the detecting a probably missing part in the received data occur in response to at least one of a group consisting of the received data having a missing value in a time series, the received data having an abnormal value, the received data having a value that does not change for a fixed period, the received data having a value whose change rate is abnormal, a data collection interval change, a correlation among a plurality of related attributes being abnormal; a value obtained by comparing differences among values obtained from repeated actions being abnormal, or referencing to missing history data in which a missing history is stored.

11. The method according to claim 1 , wherein the updating the result of the stream processing includes:
if the received data has the missing part, processing the master data so as to update a result caused from the missing part.

12. The method according to claim 1 , wherein the step of updating a result of the stream processing includes:
updating, by the processor of the computing system, a report configured to indicate an update status in real time.

13. The method according to claim 12 , wherein updating the report includes at least one of a group consisting of causing the report to indicate that the update is completed, causing the report to indicate that the data has no missing part, or causing the report to indicate a progress of the update.

14. The method according to claim 1 , wherein the updating the result of the stream processing includes:
if the updating cannot be completed within a processing time allowable for the update, stopping, by the processor, the updating process; and
marking, by the processor of the computing system, an unupdated result in response to stopping the updating process.

15. A computer system for reprocessing data having a missing part, the system comprising at least one computer configured for:
receiving, by a processor of a computing system, data;
stream-processing, by the processor of the computing system, the received data in real time;
detecting, by the processor of the computing system, a probably missing part in the received data during the stream processing by:
stream processing of the received data in a first stream pr

Assignees

• IBM

Inventors

• Akiyama Kazuhito
• Gotoh Yasuhisa
• Ogihara Hiroya

Classifications

• G06F2201/805

  Real-time · CPC title

• G06F7/02 Primary

  Comparing digital values (G06F7/06, {G06F7/22,} G06F7/38 take precedence) · CPC title

• H04L1/0084 Primary

  Formats for payload data · CPC title

• G06F11/1443

  Transmit or communication errors · CPC title

• G06F11/0751 Primary

  Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06) · CPC title