Data stream quality management for analytic environments

Patent metadata

Field	Value
Publication number	US-9460131-B2
Application number	US-201213463850-A
Country	US
Kind code	B2
Filing date	May 4, 2012
Priority date	May 4, 2012
Publication date	Oct 4, 2016
Grant date	Oct 4, 2016

Abstract

According to one aspect of the present disclosure, a system and technique for data quality management is disclosed. The system includes a processor and an ingress quality specification (IQS) module executable by the processor in a runtime environment with a data stream analytic module. The IQS module is configured to: receive the data stream; analyze a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module; annotate the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and output the data stream to the analytic module.

First claim

What is claimed is:

1. A system, comprising:
an analytic module configured to analyze a data stream output by an object and output an analysis of the object, the analytic module having a data quality expectation of data of the data stream;
a memory storing a plurality of ingress quality specification (IQS) modules each corresponding to a different quality characteristic and each associated with the analytic module; and
an interface configured to enable a selection of at least one IQS module from the plurality of IQS modules to deploy with the analytic module to:
receive the data stream from the object;
analyze a subset of data of the data stream to determine if the subset of data meets the quality expectation of the analytic module;
modify the subset of data by annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and
output the data stream to the analytic module; and

wherein the analytic module is configured to receive the data stream from the IQS module, identify data not meeting the quality expectation based on the annotations, and omit from its analysis of the object the data not meeting the quality expectation; and
wherein the selected IQS module is configured to determine whether a selected subset of data of the data stream includes a minimum quantity of data samples based on the quality expectation of the analytic module.

2. The system of claim 1 , wherein the selected IQS module is configured to:
apply a predicate to the subset of data.

3. The system of claim 2 , wherein the selected IQS module is configured to flag data of the subset passing the predicate.

4. The system of claim 3 , wherein the selected IQS module is configured to flag data of the subset failing the predicate.

5. The system of claim 1 , wherein the selected IQS module is configured to annotate a field of the data indicating the quality status.

6. The system of claim 1 , wherein the selected IQS module includes:
a selector module configured to select the subset of the data stream; and
a predicate module configured to apply a predicate to the selected subset of data to determine whether the selected subset of data meets the quality expectation of the analytic module.

7. The system of claim 6 , wherein the selector module is configured to select the subset of data based on a predetermined time period.

Assignees

• George Randy
• Mckeown Robert J
• IBM

Inventors

• George Randy
• Mckeown Robert J

Classifications

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title

• G06F16/22 Primary

  Indexing; Data structures therefor; Storage structures · CPC title