Abnormal data detection

Patent metadata

Field	Value
Publication number	US-11003739-B2
Application number	US-201916722946-A
Country	US
Kind code	B2
Filing date	Dec 20, 2019
Priority date	Jan 26, 2018
Publication date	May 11, 2021
Grant date	May 11, 2021

Abstract

This specification describes techniques for detecting abnormal data in a data set. One example method includes obtaining, by a data processing platform, a to-be-validated data group including to-be-validated data corresponding to a predetermined feature; obtaining, by the data processing platform, a comparison data group including historical data associated with the to-be-validated data group, wherein the historical and the to-be-validated data are from a same data source; performing, by the data processing platform, a two-group significance test on the to-be-validated data group and the comparison data group to generate a test result; and determining, by the data processing platform, whether there is abnormal data in the to-be-validated data group based on the test result.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
obtaining, by a data processing platform and from a second data platform, a to-be-validated data group including to-be-validated data corresponding to a predetermined set of features, wherein the second data platform collects initial data and modifies the initial data by an encryption preprocessing process that encrypts the initial data to generate the to-be-validated data;
obtaining, by the data processing platform and from a same source as the to-be-validated data group, a comparison data group including historical data values for the predetermined set of features of the to-be-validated data group; and
determining, by the data processing platform and based on performing a two-group significance test on values of the to-be-validated data group and the historical data values of the comparison data group, that there is abnormal data in the to-be-validated data group;
in response, dividing, by the data processing platform, the to-be-validated data group into a plurality of data group subsets that are each of a smaller size than that of the to-be-validated data group;
identifying, by the data processing platform, a particular data group subset that includes the abnormal data based on, for each data group subset of the plurality of data group subsets;
performing a two-group significance test on values of the data group subset and the historical data values of the comparison data group to generate a test result for the data group subset; and
determining whether there is abnormal data in the data group subset based on the test result; and

in response to identifying the particular data group subset that includes the abnormal data:
triggering an alert for initializing further analysis of the particular data group subset.

2. The computer-implemented method of claim 1 , wherein obtaining the comparison data group including the historical data includes:
obtaining a plurality of groups of historical data associated with the to-be-validated data group;
performing a two-group significance test on each of two groups of the historical data; and
determining a group of the historical data that contains no abnormal data as a comparison data group based on a test result of the two-group significance test.

3. The computer-implemented method of claim 1 , wherein the values for the predetermined set of features comprise numerical values within a predetermined range.

4. The computer-implemented method of claim 1 , wherein performing a two-group significance test includes determining a probability that a population mean associated with the to-be-validated data group is a same with a population mean associated with the comparison data group.

5. The computer-implemented method of claim 4 , wherein it is determined that there is no abnormal data in in the to-be-validated data group if the probability is greater than 0.01%.

6. The computer-implemented method of claim 1 , wherein respective data values of the to-be-validated data group and the comparison data group are not normally distributed, and wherein the method further comprises:
prior to performing the two-group significance test,
transforming data values in these two groups to have a normal distribution by performing a corresponding data transformation on the data values based on a distribution feature of the data.

7. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:
obtaining, by a data processing platform and from a second data platform, a to-be-validated data group including to-be-validated data corresponding to a predetermined set of features, wherein the second data platform collects initial data and modifies the initial data by an encryption preprocessing process that encrypts the initial data to generate the to-be-validated data;
obtaining, by the data processing platform and from a same source as the to-be-validated data group, a comparison data group including historical data values for the predetermined set of features of the to-be-validated data group; and
determining, by the data processing platform and based on performing a two-group significance test on values of the to-be-validated data group and the historical data values of the comparison data group, that there is abnormal data in the to-be-validated data group;
in response, dividing, by the data processing platform, the to-be-validated data group into a plurality of data group subsets that are each of a smaller size than that of the to-be-validated data group;
identifying, by the data processing platform, a particular data group subset that includes the abnormal data based on, for each data group subset of the plurality of data group subsets;
performing a two-group significance test on values of the data group subset and the historical data values of the comparison data group to generate a test result for the data group subset; and
determining whether there is abnormal data in the data group subset based on the test result; and

in response to identifying the particular data group subset that includes the abnormal data:
triggering an alert for initializing further analysis of the particular data group subset.

8. The non-transitory, computer-readable medium of claim 7 , wherein obtaining the comparison data group including the historical data includes:
obtaining a plurality of groups of historical data associated with the to-be-validated data group;
performing a two-group significance test on each of two groups of the historical data; and
determining a group of the historical data that contains no abnormal data as a comparison data group based on a test result of the two-group significance test.

9. The non-transitory, computer-readable medium of claim 7 , wherein the values for the predetermined set of features comprise numerical values within a predetermined range.

10. The non-transitory, computer-readable medium of claim 7 , wherein performing a two-group significance test includes determining a probability that a population mean associated with the to-be-validated data group is a same with a population mean associated with the comparison data group.

11. The non-transitory, computer-readable medium of claim 10 , wherein it is determined that there is no abnormal data in in the to-be-validated data group if the probability is greater than 0.01%.

12. The non-transitory, computer-readable medium of claim 7 , wherein respective data values of the to-be-validated data group and the comparison data group are not normally distributed, and wherein the operations further comprise:
prior to performing the two-group significance test,
transforming data values in these two groups to have a normal distribution by performing a corresponding data transformation on the data values based on a distribution feature of the data.

13. A computer-implemented system, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising:
obtaining, by a data processing platform and from a second data platform, a to-be-validated data group including to-be-validated data corresponding to a predetermined set of features, wherein the second data platform collects initial data and modifies the initial data by an encryption preprocessing process that encrypts the initial data to generate the to-be-validated data;
obtaining, by the data processing platform and

Assignees

• Advanced New Technologies Co Ltd

Inventors

• Li Longfei

Classifications

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F17/18 Primary

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title

• G06F18/2433

  Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F11/0751 Primary

  Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06) · CPC title