Data stream quality management for analytic environments

Patent metadata

Field	Value
Publication number	US-2017286462-A1
Application number	US-201715624295-A
Country	US
Kind code	A1
Filing date	Jun 15, 2017
Priority date	May 4, 2012
Publication date	Oct 5, 2017
Grant date	—

Abstract

Data stream quality management for analytic environments includes deploying, into a runtime environment upstream from an analytic module, an ingress quality specification (IQS) module. The IQS module receives a data stream and analyzes a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module. The subset of data is annotated to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module. The data stream is output with the annotated subset of data to the analytic module, and the analytic module analyzes the data stream to assess an operating characteristic of an upstream device.

First claim

What is claimed is:

1 . A method, comprising:
deploying, into a runtime environment upstream from an analytic module, an ingress quality specification (IQS) module; receiving, by the IQS module, a data stream; analyzing, by the IQS module, a subset of data of the data stream to determine if the subset of data meets a quality expectation of the analytic module; annotating the subset of data to indicate a quality status based on whether the subset of data meets the quality expectation of the analytic module; and outputting the data stream with the annotated subset of data to the analytic module, the analytic module analyzing the data stream to assess an operating characteristic of an upstream device.

2 . The method of claim 1 , further comprising receiving a selection of the IQS module from a plurality of IQS modules associated with the analytic module.

3 . The method of claim 1 , further comprising:
selecting, by the IQS module, a portion of the data stream as the subset of data; and applying, by the IQS module, a predicate to the subset of data.

4 . The method of claim 3 , further comprising flagging data of the subset passing the predicate.

5 . The method of claim 4 , further comprising flagging data of the subset failing the predicate.

6 . The method of claim 1 , further comprising determining, by the IQS module, whether the selected subset of data of the data stream includes a minimum quantity of data samples based on the quality expectation of the analytic module.

7 . The method of claim 1 , wherein annotating comprises annotating the subset of data via an additional tuple field of the data.

8 . A method, comprising:
deploying, into a runtime environment upstream from an analytic module, an ingress quality specification (IQS) module; receiving, by the IQS module, a data stream; selecting, by the IQS module, a subset of data of the data stream; determining if the subset of data meets a quality expectation of the analytic module by analyzing whether the subset of data includes a minimum quantity of data samples based on the quality expectation of the analytic module; responsive to determining that the subset of data meets the quality expectation, annotating the subset of data to indicate a quality status; and outputting the data stream with the annotated subset of data to the analytic module.

9 . The method of claim 8 , wherein annotating comprises flagging data of the subset meeting the quality expectation.

10 . The method of claim 9 , wherein annotating comprises flagging data of the subset failing to meet the quality expectation.

11 . The method of claim 8 , further comprising receiving, via an interface, a selection of the IQS modules to deploy with the analytic module.

12 . The method of claim 8 , further comprising:
independently deploying the IQS module relative to the analytic module; and linking the IQS module to the analytic module.

13 . The method of claim 9 , wherein the data stream comprises a first data stream received from a first source and second data stream received from a second source.

14 . A method, comprising:
receiving, by a first ingress quality specification (IQS) module deployed in a runtime environment, a data stream; applying, by the first IQS module, a first predicate to a subset of data of the data stream to determine if the subset of data meets a first quality expectation of a downstream analytic module; making a first annotation in the subset of data to indicate a quality status based on a result of applying the first predicate; outputting by the first IQS module the data stream to a second IQS module; applying, by the second IQS module, a second predicate to the subset of data to determine if the subset of data meets a second quality expectation of the analytic module; making a second annotation in the subset of data to indicate a quality status based on a result of applying the second predicate; receiving, by the analytic module, the data stream from the second IQS module and analyzing the subset of data to assess a performance level of a monitored object, the analytic module discerning between the first annotation and the second annotation to determine whether to include the subset of data in the assessment of the monitored object.

15 . The method of claim 14 , wherein making the first and second annotation comprises flagging data of the subset meeting the respective first and second predicates.

16 . The method of claim 15 , wherein making the first and second annotation further comprises flagging data of the subset failing the respective first and second predicates.

17 . The method of claim 14 , further comprising omitting, by the analytic module, the subset of data from an analysis of the data stream in response to the subset failing the first or second predicate.

18 . The method of claim 14 , wherein making the first and second annotation comprises annotating the subset of data via an additional tuple field of the data.

19 . The method of claim 14 , further comprising receiving, via an interface, a selection of the first and second IQS modules to deploy with the analytic module.

20 . The method of claim 14 , further comprising:
independently deploying the first and second IQS modules relative to the analytic module; and linking the first and second IQS modules to the analytic module.

Assignees

• IBM

Inventors

• George Randy
• Mckeown Robert J

Classifications

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06F16/22 Primary

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title