Data clean-up method for improving predictive model training

Patent metadata

Field	Value
Publication number	US-12412127-B2
Application number	US-202318453914-A
Country	US
Kind code	B2
Filing date	Aug 22, 2023
Priority date	Oct 15, 2014
Publication date	Sep 9, 2025
Grant date	Sep 9, 2025

Abstract

A method that improves the training of predictive models. Better trained predictive models make better predictions, and can classify transactions with reduced levels of false positives and false negative. Included is an apparatus for executing a data clean-up algorithm that harmonizes a wide range of real world supervised and unsupervised training data into a single, error-free, uniformly formatted record file that has every field coherent and well populated with information.

First claim

The invention claimed is:

1. A computer-implemented method for training a predictive model, the method comprising, via one or more transceivers and/or processors:
receiving a plurality of records, each of the plurality of records including a plurality of data fields and the predictive model including a smart agent corresponding to each of the plurality of data fields and a classification model including one or more of data mining logic, a neural network, case-based-reasoning, clustering or business rules;
generating a record file including a value populating each of the plurality of data fields;
executing a computer learning training algorithm to train the plurality of smart agents and the one or more classification models of the predictive model based on the record file including by—
for each of plurality of data fields that is numeric, determining at least one normal numeric value interval based on the plurality of values in the record file populating the corresponding one of the plurality of data fields,
for each of the plurality of data fields that is symbolic, determining at least one normal symbolic value based on the plurality of values in the record file populating the corresponding one of the plurality of data fields,
wherein the predictive model is configured to combine a plurality of scores output by the plurality of smart agents and the one or more classification models into a single result.

2. The computer-implemented method of claim 1 , wherein one or more of the plurality of smart agents corresponding to a numeric one of the plurality of data fields comprises a plurality of normal numeric value intervals.

3. The computer-implemented method of claim 1 , further comprising, via the one or more processors and/or transceivers and based on determining that no normal numeric value interval is computable for an additional data field of the plurality of records, configuring the predictive model to omit validity testing for a smart agent corresponding to the additional data field.

4. The computer-implemented method of claim 1 , further comprising, via the one or more processors and/or transceivers, omitting those of the values populating one or more of the plurality of data fields that are symbolic from the corresponding at least one normal symbolic value based on failure to reach a minimum cardinality threshold.

5. The computer-implemented method of claim 1 , further comprising, via the one or more processors and/or transceivers, omitting those of the values populating one or more of the plurality of data fields that are numeric from the corresponding at least one normal numeric value interval based on failure to reach a minimum cardinality threshold.

6. The computer-implemented method of claim 1 , further comprising, via the one or more processors and/or transceivers, omitting numeric intervals calculated from those of the values populating one or more of the plurality of data fields that are numeric if the omitted numeric intervals would cause a maximum number of intervals for the corresponding one of the smart agents to be exceeded.

7. The computer-implemented method of claim 1 , further comprising, via the one or more processors and/or transceivers, omitting numeric intervals calculated from those of the values populating one or more of the plurality of data fields that are numeric if those of the values corresponding to the omitted numeric intervals fail to satisfy a minimum frequency.

8. The computer-implemented method of claim 1 , wherein determining the at least one normal numeric value interval for at least one of the plurality of data fields includes applying a maximum interval distance.

9. The computer-implemented method of claim 1 , further comprising, via the one or more processors and/or transceivers—
determining a dependency between two of the plurality of data fields based at least in part on a frequency of cooccurrence of the values populating the two data fields,
configuring one of the plurality of smart agents based on the dependency if the frequency of cooccurrence meets or exceeds a threshold.

10. The computer-implemented method of claim 9 , further comprising, via the one or more processors and/or transceivers, retraining the one of the plurality of smart agents by—
calculating an updated frequency of cooccurrence by incorporating new values of the two data fields from new records,
removing the dependency based on a determination that the updated frequency of cooccurrence does not meet or exceed the threshold.

11. The computer-implemented method of claim 9 , further comprising, via the one or more processors and/or transceivers—
calculating an updated frequency of cooccurrence by incorporating new values of the two data fields from new records,
retaining the dependency based on a determination that the updated frequency of cooccurrence meets or exceeds the threshold.

12. The computer-implemented method of claim 1 , wherein generating the record file includes—
determining that a data field of the plurality of data fields includes invalid contents using a data dictionary,
revising the record file at least in part by substituting a replacement data value for the invalid contents of the data field.

13. The computer-implemented method of claim 12 , wherein the data dictionary is a numeric data dictionary, further comprising—
determining that the data field is a numeric data field based at least in part on detecting a corresponding numeric data field type;
accessing, in connection with determination that the contents of the data field are invalid, the numeric data dictionary based at least in part on the determination of the numeric data field type.

14. The computer-implemented method of claim 13 , wherein determining that the contents of the data field are invalid includes determining either: (A) that the contents are not numeric, or (B) that the contents are not within a numeric range.

15. The computer-implemented method of claim 12 , wherein the data dictionary is a symbolic data dictionary, further comprising—
determining that the data field is a symbolic data field based at least in part on detecting a corresponding symbolic data field type;
accessing, in connection with determination that the contents of the data field are invalid, the symbolic data dictionary based at least in part on the determination of the symbolic data field type.

16. The computer-implemented method of claim 15 , wherein determining that the contents of the data field are invalid includes determining one or more of the following: (A) that the data field is empty; (B) that the data field includes contents comprising values not allowed in a set; (C) that the data field is of a zipcode type and includes contents comprising a zipcode that is not valid; (D) that the data field is of a telephone or fax type and includes contents comprising a telephone number or fax number that is not valid; and (E) that the data field is of a date/time field type and does not contain at least one of a valid date and time.

17. The computer-implemented method of claim 12 , further comprising determining, via the one or more processors, the replacement data value according to at least one of the following: (A) a default value; (B) an average value; (C) a minimum value; (D) a maximum value; and (E) a null value.

18. The computer-implemented method of claim 12 , wherein the replacement data value is a replacement symbolic data value, further comprising determining, via the one or more processors, the replacement symbolic data value according to at least one of the foll

Assignees

• Brighterion Inc

Inventors

• Adjaoute Akli

Classifications

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N20/00 Primary

  Machine learning · CPC title