Managing defects in a model training pipeline using synthetic data sets associated with defect types

Patent metadata

Field	Value
Publication number	US-2021406727-A1
Application number	US-202016917769-A
Country	US
Kind code	A1
Filing date	Jun 30, 2020
Priority date	Jun 30, 2020
Publication date	Dec 30, 2021
Grant date	—

Abstract

The disclosure herein describes managing defects in a model training pipeline. A synthetic data set is generated that is associated with a defect type and a lifecycle stage of the model training pipeline, and baseline performance metrics associated with the defect type are generated. Based on a code change to the pipeline, a test model is trained using the pipeline and the synthetic data set, and test performance metrics are collected based on the test model and associated with the defect type. Based on comparing the baseline performance metrics and the test performance metrics, a defect of a particular defect type is identified in the pipeline. An indicator of the defect is provided that includes the defect type and the lifecycle stage with which the synthetic data set is associated, whereby a defect correction process is enabled to remedy the defect based on the associated defect type and the lifecycle stage.

First claim

1 . A system for managing defects in a model training pipeline, the system comprising:
at least one processor; and at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the at least one processor to: generate a plurality of synthetic data sets, wherein each synthetic data set of the plurality of synthetic data sets is associated with a defect type of a plurality of defect types and an associated lifecycle stage of a plurality of lifecycle stages in the model training pipeline; generate baseline performance metrics based on each synthetic data set of the plurality of synthetic data sets and associated with each defect type of the plurality of defect types; based on a code change being made to the model training pipeline, train a plurality of test models using the model training pipeline and the plurality of synthetic data sets, wherein a test model of the plurality of test models is trained for each synthetic data set of the plurality of synthetic data sets; based on the training of the plurality of test models using the model training pipeline, collect test performance metrics based on each test model of the plurality of test models and associated with each defect type of the plurality of defect types; based on comparing the baseline performance metrics to the test performance metrics, identify at least one defect of at least one defect type of the plurality of defect types in the model training pipeline; and provide at least one indicator of the identified at least one defect, wherein each indicator of the at least one indicator includes the defect type and the lifecycle stage with which the defect of the indicator is associated, whereby a defect correction process is enabled to remedy the at least one defect based on the associated defect type and lifecycle stage.

2 . The system of claim 1 , wherein generating the baseline performance metrics based on each synthetic data set of the plurality of synthetic data sets and associated with each defect type of the plurality of defect types includes, for each synthetic data set of the plurality of synthetic data sets:
training a baseline model using a baseline version of the model training pipeline and the synthetic data set; and based on the training of the baseline model using the model training pipeline, collecting the baseline performance metrics using the baseline model, wherein the collected baseline performance metrics are associated with the defect type and lifecycle stage of the synthetic data set.

3 . The system of claim 2 , wherein the baseline performance metrics include a baseline model accuracy percentage for each baseline model and the test performance metrics include a test model accuracy percentage for each test model of the plurality of test models;
wherein comparing the baseline performance metrics and the test performance metrics includes comparing a difference between the baseline model accuracy percentage and the test model accuracy percentage to a threshold value for each test model of the plurality of test models; and wherein identifying that at least one defect of the at least one defect type is present in the model training pipeline is further based on the difference between the baseline model accuracy percentage and the test model accuracy percentage exceeding the threshold value and the test model accuracy percentage being less than the baseline model accuracy percentage for at least one test model of the plurality of test models.

4 . The system of claim 1 , wherein generating the plurality of synthetic data sets includes, for each defect type of the plurality of defect types, transforming each real data set of a plurality of real data sets to include a data pattern associated with the defect type, whereby a defect of the defect type in the model training pipeline causes performance of models trained using the model training pipeline and the transformed data set with the data pattern to degrade from the baseline performance metrics.

5 . The system of claim 1 , wherein the plurality of lifecycle stages includes at least one of a data quality stage, a featurization stage, a sampling stage, or a model training stage.

6 . The system of claim 1 , wherein the plurality of defect types include at least one of an imputation defect type, an outlier handling defect type, a featurization defect type, a skewed predictor defect type, a stratified sample defect type, or a machine learning algorithm-specific defect type.

7 . The system of claim 1 , wherein the at least one memory and the computer program code are configured to, with the at least one processor, further cause the processor to:
based on comparing the baseline performance metrics to the test performance metrics and the test performance metrics exceeding the baseline performance metrics, provide an indication that the code change improves the model training pipeline with respect to at least one defect type of the plurality of defect types, whereby a baseline version of the model training pipeline is updated to include the code change and the baseline performance metrics are updated based on the test performance metrics.

8 . A computerized method for managing defects in a model training pipeline, the computerized method comprising:
generating, by a processor, a synthetic data set, wherein the synthetic data set is associated with a defect type and a lifecycle stage in the model training pipeline; generating, by the processor, baseline performance metrics associated with the defect type based on the generated synthetic data set; based on a code change being made to the model training pipeline, training a test model, by the processor, using the model training pipeline and the generated synthetic data set; based on the training of the test model using the model training pipeline, collecting, by the processor, test performance metrics based on the test model and associated with the defect type; based on comparing the baseline performance metrics to the test performance metrics, identifying, by the processor, a defect of the defect type in the model training pipeline; and providing, by the processor, an indicator of the identified defect including the defect type and the lifecycle stage with which the generated synthetic data set is associated, whereby a defect correction process is enabled to remedy the identified defect based on the associated defect type and the lifecycle stage.

9 . The computerized method of claim 8 , wherein generating the baseline performance metrics associated with the defect type based on the generated synthetic data set includes:
training a baseline model, by the processor, using a baseline version of the model training pipeline and the generated synthetic data set; and based on the training of the baseline model using the model training pipeline, collecting, by the processor, the baseline performance metrics using on the baseline model, wherein the collected baseline performance metrics are associated with the defect type and lifecycle stage of the generated synthetic data set.

10 . The computerized method of claim 9 , wherein the baseline performance metrics include a baseline model accuracy percentage and the test performance metrics include a test model accuracy percentage;
wherein comparing the baseline performance metrics and the test performance metrics includes comparing a difference between the baseline model accuracy percentage and the test model accuracy percentage to a threshold value; and wherein identifying the defect of the defect type in the model training pipeline is further based on the difference between the baseline model accuracy p

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Wang Shoou-Jiun
• Zhang Xing
• Abdelreheem Eslam K

Classifications

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title