Stream-processing data

Patent metadata

Field	Value
Publication number	US-10579453-B2
Application number	US-201816179010-A
Country	US
Kind code	B2
Filing date	Nov 2, 2018
Priority date	Sep 29, 2014
Publication date	Mar 3, 2020
Grant date	Mar 3, 2020

Abstract

A method for stream-processing data including a missing part in real time and thereafter updating the result of the stream processing. A technique for processing data is included. The technique includes receiving data; detecting a probably missing part in the received data while stream-processing the received data in real time; and comparing master data corresponding to the received data and having no missing part with the probably missing part, and if the received data has the missing part, updating the result of the stream processing using the master data.

First claim

What is claimed is:

1. A method comprising:
processing received time-series data in real time in a first stream processing line to produce a stream processing result, wherein the first stream processing line comprises a combination of a plurality of operators, each operator of the plurality of operators being one processing unit executed by a processor of a computing system to perform a process on the received time-series data;
determining that data with a missing value is present in the received time-series data;
relocating, by the processor of the computing system, the plurality of operators to define a second stream processing line;
calculating an amount of a system resource required for execution of the relocated operators in the second stream processing line;
in response to determining that the calculated amount of the system resource exceeds a first threshold, reducing the calculated amount of the system resource by deleting a portion of modules of a particular operator of the plurality of relocated operators to modify the second stream processing line; and
updating the stream processing result using the modified second stream processing line.

2. The method of claim 1 , wherein relocation of the plurality of operators is determined in accordance with an order of priority of the stream processing result.

3. The method of claim 1 , wherein relocation of the plurality of operators is determined in accordance with a dependency relationship among operators of the combination of the plurality of operators.

4. The method of claim 1 , wherein relocation of the plurality of operators is determined in accordance with a processing time allowable for updating the stream processing result.

5. The method of claim 1 , wherein relocation of the plurality of operators is determined in accordance with a system resource available for updating the stream processing result.

6. The method of claim 1 , wherein determining that data with a missing value is present is based, at least in part, on detecting a blank in the time series data.

7. The method of claim 1 , wherein determining that data with a missing value is present in the received time-series data includes comparing the received time-series data with master data that corresponds to the received time-series data and that has no missing value.

8. A computer system comprising:
a communication interface configured to receive time-series data; and
at least one processor configured to implement a first stream processing line to process the time-series data in real time to produce a stream processing result, wherein the first stream processing line comprises a combination of a plurality of operators, each operator of the plurality of operators being one processing unit executed by the at least one processor to perform a process on the received time-series data;
wherein the at least one processor is further configured to:
determine that data with a missing value is present in the received time-series data;
relocate the plurality of operators to define a second stream processing line;
calculate an amount of a system resource required for execution of the relocated operators in the second stream processing line;
in response to determining that the calculated amount of the system resource exceeds a first threshold, reduce the calculated amount of the system resource by deleting a portion of modules of a particular operator of the plurality of relocated operators to modify the second stream processing line; and
update the stream processing result using the modified second stream processing line.

9. The computer system of claim 8 , wherein the at least one processor is configured to determine relocation of the plurality of operators in accordance with an order of priority of the stream processing result.

10. The computer system of claim 8 , wherein the at least one processor is configured to determine relocation of the plurality of operators in accordance with a dependency relationship among operators of the combination of the plurality of operators.

11. The computer system of claim 8 , wherein the at least one processor is configured to determine relocation of the plurality of operators in accordance with a processing time allowable for updating the stream processing result.

12. The computer system of claim 8 , wherein the at least one processor is configured to determine relocation of the plurality of operators in accordance with a system resource available for updating the stream processing result.

13. The computer system of claim 8 , wherein the at least one processor is configured to determine that the data with the missing value is present based, at least in part, on detecting a blank in the time series data.

14. The computer system of claim 8 , wherein the at least one processor is configured to determine that the data with the missing value is present in the received time-series data by comparing the received time-series data with master data that corresponds to the received time-series data and that has no missing value.

15. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed by a processor, causes the processor to:
process received time-series data in real time in a first stream processing line to produce a stream processing result, wherein the first stream processing line comprises a combination of a plurality of operators, each operator of the plurality of operators being one processing unit executed by the processor to perform a process on the received time-series data;
determine that data with a missing value is present in the received time-series data;
relocate the plurality of operators to define a second stream processing line;
calculate an amount of a system resource required for execution of the relocated operators in the second stream processing line;
in response to determining that the calculated amount of the system resource exceeds a first threshold, reduce the calculated amount of the system resource by deleting a portion of modules of a particular operator of the plurality of relocated operators to modify the second stream processing line; and
update the stream processing result using the modified second stream processing line.

16. The computer program product of claim 15 , wherein the computer readable program is further configured to cause the processor to determine relocation of the plurality of operators in accordance with an order of priority of the stream processing result.

17. The computer program product of claim 15 , wherein the computer readable program is further configured to cause the processor to determine relocation of the plurality of operators in accordance with a dependency relationship among operators of the combination of the plurality of operators.

18. The computer program product of claim 15 , wherein the computer readable program is further configured to cause the processor to determine relocation of the plurality of operators in accordance with a processing time allowable for updating the stream processing result.

19. The computer program product of claim 15 , wherein the computer readable program is further configured to cause the processor to determine that the data with the missing value is present based, at least in part, on detecting a blank in the time series data.

20. The computer program product of claim 15 , wherein the computer readable program is further configured to cause the

Assignees

• IBM

Inventors

• Akiyama Kazuhito
• Gotoh Yasuhisa
• Ogihara Hiroya

Classifications

• G06F2201/805

  Real-time · CPC title

• G06F7/02

  Comparing digital values (G06F7/06, {G06F7/22,} G06F7/38 take precedence) · CPC title

• G06F11/1443

  Transmit or communication errors · CPC title

• H04L1/0084 Primary

  Formats for payload data · CPC title

• G06F11/0751 Primary

  Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06) · CPC title