Data stream quality management for analytic environments

Patent metadata

Field	Value
Publication number	US-10671580-B2
Application number	US-201615142505-A
Country	US
Kind code	B2
Filing date	Apr 29, 2016
Priority date	May 4, 2012
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

Data stream quality management for analytic environments includes an ingress quality specification (IQS) module with a data stream analytic module. The analytic module has a data quality expectation for data of the data stream and receives the data stream from the IQS module. A selector module receives the data stream from an object and selects a subset of the received data stream. A predicate module applies a predicate to the subset to determine if the subset meets the quality expectation of the analytic module. The predicate module modifies the subset of data by annotating the subset to indicate a quality status based on whether the subset meets the quality expectation. The IQS module outputs the data stream with the annotated subset of data to the analytic module. The analytic module outputs an analysis of the object based on whether the annotated subset of data met the quality expectation.

First claim

What is claimed is:

1. A system, comprising:
a processor;
an ingress quality specification (IQS) module executable by the processor in a runtime environment with a data stream analytic module, the data stream analytic module having a data quality expectation for data of a data stream and configured to receive the data stream from the IQS module, the IQS module including:
a selector module configured to receive the data stream from an object and select a subset of the received data stream; and
a predicate module configured to:
apply a predicate to the subset of data to analyze the subset of data to determine if the subset of data meets the quality expectation of the analytic module, the predicate determining whether the subset of data has missing data records; and
modify the subset of data by annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and

the IQS module operable to output the data stream with the annotated subset of data to the analytic module, the analytic module configured to assess an operating characteristic of the object based on whether the annotated subset of data met the quality expectation.

2. The system of claim 1 , wherein the analytic module is executable by the processor to:
receive the data stream from the IQS module;
determine the quality status of the subset of data; and
responsive to determining that the quality status indicates that the subset of data does not meet the quality expectation, omit the subset of data from an analysis of the data stream by the analytic module.

3. The system of claim 1 , wherein the IQS module is executable by the processor to flag data of the subset passing the predicate.

4. The system of claim 3 , wherein the IQS module is executable by the processor to flag data of the subset failing the predicate.

5. The system of claim 1 , wherein the processor is configured to deploy the IQS module responsive to receiving a selection of the IQS module from a plurality of IQS modules associated with the analytic module.

6. A computer program product for data quality management, the computer program product comprising:
a non-transitory computer readable medium having computer readable program code embodied therewith, the computer readable program code comprising computer readable program code configured to:
receive a data stream from an object in a runtime environment;
apply a predicate to a subset of data of the data stream to determine if the subset of data meets a quality expectation of a downstream analytic module, the predicate determining whether the subset of data has missing data records;
analyze the subset of data based on the predicate;
annotate the subset of data via an additional field of the data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and
output the data stream with the annotated subset of data to the analytic module, the analytic module operable to assess an operating characteristic of the object based on whether the annotated subset of data met the quality expectation.

7. The computer program product of claim 6 , wherein the computer readable program code is configured to:
select a portion of the data stream as the subset of data; and
apply a predicate to the subset of data.

8. The computer program product of claim 7 , wherein the computer readable program code is configured to flag data of the subset passing the predicate.

9. The computer program product of claim 8 , wherein the computer readable program code is configured to flag data of the subset failing the predicate.

10. A system, comprising:
an analytic module configured to analyze a data stream output by an object in a runtime environment and assess an operating characteristic of the object, the analytic module having a data quality expectation of data of the data stream;
a memory storing a plurality of ingress quality specification (IQS) modules each corresponding to a different quality characteristic and each associated with the analytic module; and
an interface configured to enable a selection of at least one IQS module from the plurality of IQS modules to deploy with the analytic module to:
receive the data stream from the object;
apply a predicate to a subset of data of the data stream to determine if the subset of data meets the quality expectation of the analytic module, the predicate determining whether the subset of data has missing data records;
analyze the subset of data based on the predicate;
modify the subset of data by annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and
output the data stream to the analytic module; and

wherein the analytic module is configured to receive the data stream from the IQS module, identify data not meeting the quality expectation based on the annotations, and omit from its assessment of the object the data not meeting the quality expectation.

11. The system of claim 10 , wherein the selected IQS module is configured to:
select a portion of the data stream as the subset of data; and
apply a predicate to the subset of data.

12. The system of claim 11 , wherein the selected IQS module is configured to flag data of the subset passing the predicate.

13. The system of claim 12 , wherein the selected IQS module is configured to flag data of the subset failing the predicate.

14. The system of claim 10 , wherein the selected IQS module is configured to annotate a field of the data indicating the quality status.

15. The system of claim 10 , wherein the selected IQS module comprises:
a selector module configured to select a subset of the data stream; and
a predicate module configured to apply a predicate to the selected subset of data to determine whether the selected subset of data meets the quality expectation of the analytic module.

16. The system of claim 15 , wherein the selector module is configured to select the subset of data based on a predetermined time period.

17. The system of claim 16 , wherein the predicate module is configured to determine whether the selected subset of data includes a minimum quantity of data samples based on the quality expectation of the analytic module.

Assignees

• IBM

Inventors

• George Randy
• Mckeown Robert J

Classifications

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title

• G06F16/22 Primary

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title