Data stream quality management for analytic environments

Patent metadata

Field	Value
Publication number	US-10803032-B2
Application number	US-201715624295-A
Country	US
Kind code	B2
Filing date	Jun 15, 2017
Priority date	May 4, 2012
Publication date	Oct 13, 2020
Grant date	Oct 13, 2020

Abstract

Data stream quality management for analytic environments includes deploying, into a runtime environment upstream from an analytic module, an ingress quality specification (IQS) module. The IQS module receives a data stream and analyzes a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module. The subset of data is annotated to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module. The data stream is output with the annotated subset of data to the analytic module, and the analytic module analyzes the data stream to assess an operating characteristic of an upstream device.

First claim

What is claimed is:

1. A method, comprising:
deploying, into a runtime environment upstream from an analytic module, an ingress quality specification (IQS) module;
receiving, by the IQS module, a data stream from an upstream device in the runtime environment;
analyzing, by the IQS module, a subset of data of the data stream based on a quality expectation of the data stream by the analytic module;
annotating, by the IQS module, the subset of data to indicate a quality status of the subset of data based on the analysis performed by the IQS module; and
outputting the data stream with the annotated subset of data to the analytic module, the analytic module identifying the quality status of the subset of data based on the annotation and assessing an operating characteristic of the upstream device based on the annotated subset of data.

2. The method of claim 1 , further comprising receiving a selection of the IQS module from a plurality of IQS modules associated with the analytic module.

3. The method of claim 1 , further comprising:
selecting, by the IQS module, a portion of the data stream as the subset of data; and
applying, by the IQS module, a predicate to the subset of data.

4. The method of claim 3 , further comprising flagging data of the subset passing the predicate.

5. The method of claim 4 , further comprising flagging data of the subset failing the predicate.

6. The method of claim 1 , further comprising determining, by the IQS module, whether the selected subset of data of the data stream includes a minimum quantity of data samples based on the quality expectation of the analytic module.

7. The method of claim 1 , wherein annotating comprises annotating the subset of data via an additional tuple field of the data.

8. A method, comprising:
deploying, into a runtime environment upstream from an analytic module, an ingress quality specification (IQS) module;
receiving, by the IQS module, a data stream from an upstream device within the runtime environment;
selecting, by the IQS module, a subset of data of the data stream;
analyzing, by the IQS module, the subset of data based on a quality expectation of the subset of data by the analytic module, the quality expectation indicating a minimum quantity of data samples;
annotating, by the IQS module, the subset of data to indicate a quality status based on the analysis by the IQS module relative to the quality expectation; and
outputting the data stream with the annotated subset of data to the analytic module, the analytic module identifying the quality status of the subset of data based on the annotation and assessing an operating characteristic of the upstream device based on the annotated subset of data.

9. The method of claim 8 , wherein annotating comprises flagging data of the subset meeting the quality expectation.

10. The method of claim 9 , wherein annotating comprises flagging data of the subset failing to meet the quality expectation.

11. The method of claim 8 , further comprising receiving, via an interface, a selection of the IQS modules to deploy with the analytic module.

12. The method of claim 8 , further comprising:
independently deploying the IQS module relative to the analytic module; and
linking the IQS module to the analytic module.

13. The method of claim 9 , wherein the data stream comprises a first data stream received from a first source and second data stream received from a second source.

14. A method, comprising:
deploying, in a runtime environment, first and second ingress quality specification (IQS) modules;
receiving, by the first IQS, a data stream associated with a monitored object located upstream from the first IQS module in the runtime environment;
applying, by the first IQS module, a first predicate to a subset of data of the data stream to analyze the subset of data based on a first quality expectation of a downstream analytic module;
making a first annotation in the subset of data to indicate a quality status based on a result of applying the first predicate;
outputting by the first IQS module the data stream to the second IQS module;
applying, by the second IQS module, a second predicate to the subset of data to analyze the subset of data based on a second quality expectation of the analytic module;
making a second annotation in the subset of data to indicate a quality status based on a result of applying the second predicate; and
receiving, by the analytic module, the data stream from the second IQS module and analyzing the subset of data to assess a performance level of a monitored object, the analytic module discerning between the first annotation and the second annotation to identify a quality status of the subset of data based on the application of the respective first and second predicates.

15. The method of claim 14 , wherein making the first and second annotation comprises flagging data of the subset meeting the respective first and second predicates.

16. The method of claim 15 , wherein making the first and second annotation further comprises flagging data of the subset failing the respective first and second predicates.

17. The method of claim 14 , further comprising omitting, by the analytic module, the subset of data from an analysis of the data stream in response to the subset failing the first or second predicate.

18. The method of claim 14 , wherein making the first and second annotation comprises annotating the subset of data via an additional tuple field of the data.

19. The method of claim 14 , further comprising receiving, via an interface, a selection of the first and second IQS modules to deploy with the analytic module.

20. The method of claim 14 , further comprising:
independently deploying the first and second IQS modules relative to the analytic module; and
linking the first and second IQS modules to the analytic module.

Assignees

• IBM

Inventors

• George Randy
• Mckeown Robert J

Classifications

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F16/22 Primary

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title