Systems and methods for data quality control and cleansing

Patent metadata

Field	Value
Publication number	US-9354968-B2
Application number	US-201213631301-A
Country	US
Kind code	B2
Filing date	Sep 28, 2012
Priority date	Sep 30, 2011
Publication date	May 31, 2016
Grant date	May 31, 2016

Abstract

A method for detecting and cleansing suspect building automation system data is shown and described. The method includes using processing electronics to automatically determine which of a plurality of error detectors and which of a plurality of data cleansers to use with building automation system data. The method further includes using processing electronics to automatically detect errors in the data and cleanse the data using a subset of the error detectors and a subset of the cleansers.

First claim

We claim:

1. A method for detecting and cleansing suspect building automation system data, comprising:
using processing electronics to automatically determine which of a plurality of error detectors and which of a plurality of data cleansers to use with building automation system data;
using the processing electronics to automatically detect errors in the data and cleanse the data using a subset of the error detectors and a subset of the cleansers;
wherein the processing electronics automatically determine which of the plurality of data cleansers to use with the building automation system data prior to cleansing the data;
flagging a data point as suspect data if an absolute value of a rate of change in data value between the data point and a previous data point is greater than a critical derivative; and
using a growth function to either:
(a) increase the critical derivative if the absolute value of the rate of change in data value between the data points is greater than the critical derivative, or
(b) decrease the critical derivative if the absolute value of the rate of change in data value between the data points is less than or equal to the critical derivative.

2. The method of claim 1 , wherein the processing electronics use an indication of data type to determine which of the error detectors or which of the data cleansers to use with the data.

3. The method of claim 1 , wherein the processing electronics use information concerning how the data will be used in subsequent processing to determine which of the error detectors or which of the data cleansers to use with the data.

4. The method of claim 1 , wherein the processing electronics use information concerning which of the error detectors is used with the data or the type of error determined by the detectors to determine which of the data cleansers to use with the data.

5. The method of claim 1 , further comprising:
using the processing electronics to generate a message for transmission to a user, wherein the message for transmission to the user indicates the presence of a detected error and the type of the detected error.

6. The method of claim 1 , wherein the plurality of cleansers comprise at least two of:
(a) a module that replaces the suspect data with a not-a-number value;
(b) a module that replaces the suspect data with a value determined by interpolation;
(c) a module that formats the data to a uniform format; and
(d) a module that sorts the data.

7. The method of claim 1 , wherein the plurality of error detectors comprise at least three of:
(a) a static bounds error detector;
(b) an adaptive bounds error detector;
(c) a static derivative bounds error detector;
(d) an adaptive derivative bounds error detector; and
(e) a stuck value error detector.

8. The method of claim 7 , wherein the static bounds error detector uses a process comprising:
receiving a data point having a data value;
receiving a minimum and maximum bound information; and
flagging the data point as suspect data if the data value is not within the minimum and maximum bounds.

9. The method of claim 7 , wherein the adaptive bounds error detector uses a process comprising:
receiving a critical value and a data window having a window size;
receiving data points until the number of data points collected equals the window size;
estimating a data spread and a central tendency for the data points in the current data window;
receiving a next data point having a data value; and
flagging the next data point as suspect data if the absolute value of the difference between the data value and the central tendency is greater than the product of the critical value and the data spread.

10. The method of claim 7 , wherein the static derivative bounds error detector uses a process comprising:
receiving a critical derivative, an initial data point and a next data point, each data point having a data value and a data timestamp;
determining a rate of change in data value between the initial data point and the next data point; and
flagging the next data point as suspect data if the rate of change is greater than the critical derivative.

11. The method of claim 7 , wherein the adaptive derivative bounds error detector uses a process comprising:
receiving a critical derivative and a growth function;
collecting an initial data point having an initial data value and an initial data timestamp and a next data point having a next data value and a next data timestamp;
determining the rate of change in data value between the data points by dividing the difference in data values by the difference in data timestamps;
flagging the next data point as suspect data if the absolute value of the rate of change in data value between the data points is greater than the critical derivative; and
using the growth function to either
(a) increase the critical derivative if the absolute value of the rate of change in data value between the data points is greater than the critical derivative, or
(b) decrease the critical derivative if the absolute value of the rate of change in data value between the data points is less than or equal to the critical derivative.

12. The method of claim 7 , wherein the stuck value error detector uses a process comprising:
receiving a maximum false alarm rate and a plurality of data points, each data point having a data value and a data timestamp;
determining an average time between changes in data value;
determining a first critical time based on the average time between changes and the maximum false alarm rate;
receiving a next data point having a next data value and a next data timestamp;
determining a difference between the next data timestamp and the timestamp of the most recent previous data point in the plurality of data points with a data value different from the next data value; and
flagging the next data point as stuck if the difference between the next data timestamp and the timestamp of the most recent previous data point with a data value different from the next data value is greater than the first critical time.

13. The method of claim 12 , wherein the stuck value error detector uses a process further comprising:
determining a current noise estimate and using the current noise estimate to calculate a noise band;
determining an average time between deviations from the noise band;
determining a second critical time based on the average time between deviations from the noise band and the maximum false alarm rate;
determining a difference between the next data timestamp and the timestamp of the most recent previous data point in the plurality of data points with a data value significantly different from the next data value; and
flagging the next data point as stuck if the difference between the next data timestamp and the timestamp of the most recent previous data point with a data value significantly different from the next data value is greater than the second critical time, wherein a difference in data value between two data points is significantly different if the difference represents a deviation from the noise band.

14. A computerized method for detecting suspect building automation system data, the method comprising:
receiving, at a processing circuit of an adaptive bounds error detector, a critical value and a data window having a window size;
collecting, at the processing circuit, data points until the number of data points collected equals the window size;
using robust statistics to estimate, by the processing circuit, data spread and central tendency in the current data window;
collecting a next data point having a data value;

Assignees

• Johnson Controls Tech Co

Inventors

• Wenzel Michael J
• Boettcher Andrew J
• Drees Kirk H
• Kummer James P

Classifications

• G06Q10/06

  Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F11/0793 Primary

  Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40) · CPC title

• G06F11/076

  by exceeding a count or rate limit, e.g. word- or bit count limit · CPC title

• G06F11/0727

  in a storage system, e.g. in a DASD or network based storage system (drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097) · CPC title