Managing defects in a model training pipeline using synthetic data sets associated with defect types

Patent metadata

Field	Value
Publication number	US-11580425-B2
Application number	US-202016917769-A
Country	US
Kind code	B2
Filing date	Jun 30, 2020
Priority date	Jun 30, 2020
Publication date	Feb 14, 2023
Grant date	Feb 14, 2023

Abstract

The disclosure herein describes managing defects in a model training pipeline. A synthetic data set is generated that is associated with a defect type and a lifecycle stage of the model training pipeline, and baseline performance metrics associated with the defect type are generated. Based on a code change to the pipeline, a test model is trained using the pipeline and the synthetic data set, and test performance metrics are collected based on the test model and associated with the defect type. Based on comparing the baseline performance metrics and the test performance metrics, a defect of a particular defect type is identified in the pipeline. An indicator of the defect is provided that includes the defect type and the lifecycle stage with which the synthetic data set is associated, whereby a defect correction process is enabled to remedy the defect based on the associated defect type and the lifecycle stage.

First claim

What is claimed is:

1. A system for managing a defect in a model training pipeline, the system comprising:
a processor; and
a memory comprising computer program code, the memory and the computer program code configured to, with the processor, cause the processor to:
generate a plurality of synthetic data sets, wherein each synthetic data set of the plurality of synthetic data sets is associated with a defect type of a plurality of defect types and an associated lifecycle stage of a plurality of lifecycle stages in the model training pipeline;
generate baseline performance metrics based on each synthetic data set of the plurality of synthetic data sets, wherein each baseline performance metric is associated with a defect type of the plurality of defect types;
based on a code change being made to the model training pipeline, train a plurality of test models using the model training pipeline and the plurality of synthetic data sets, wherein a test model of the plurality of test models is trained for each synthetic data set of the plurality of synthetic data sets; and
based on comparing the baseline performance metrics to test performance metrics, identify at least one defect of at least one defect type of the plurality of defect types in the model training pipeline.

2. The system of claim 1 , wherein generating the baseline performance metrics based on each synthetic data set of the plurality of synthetic data sets includes, for each synthetic data set of the plurality of synthetic data sets:
training a baseline model using a baseline version of the model training pipeline and the synthetic data set of the plurality of synthetic data sets; and
based on the training of the baseline model using the model training pipeline, collecting the baseline performance metrics using the baseline model, wherein the collected baseline performance metrics are associated with the defect type of the synthetic data set and lifecycle stage of the synthetic data set.

3. The system of claim 2 , wherein the baseline performance metrics include a baseline model accuracy percentage for each baseline model and the test performance metrics include a test model accuracy percentage for each test model of the plurality of test models;
wherein comparing the baseline performance metrics and the test performance metrics includes comparing a difference between the baseline model accuracy percentage and the test model accuracy percentage to a threshold value for each test model of the plurality of test models; and
wherein identifying that at least one defect of the at least one defect type is present in the model training pipeline is further based on the difference between the baseline model accuracy percentage and the test model accuracy percentage exceeding the threshold value and the test model accuracy percentage being less than the baseline model accuracy percentage for at least one test model of the plurality of test models.

4. The system of claim 1 , wherein generating the plurality of synthetic data sets includes, for each defect type of the plurality of defect types, transforming each real data set of a plurality of real data sets to include a data pattern associated with the defect type, whereby a defect of the defect type in the model training pipeline causes performance of models trained using the model training pipeline and the transformed real data set with the data pattern to degrade from the baseline performance metrics.

5. The system of claim 1 , wherein the plurality of defect types include at least one of an imputation defect type, an outlier handling defect type, a featurization defect type, a skewed predictor defect type, a stratified sample defect type, or a machine learning algorithm-specific defect type.

6. The system of claim 1 , wherein the memory and the computer program code are configured to, with the processor, further cause the processor to:
based on comparing the baseline performance metrics to the test performance metrics and the test performance metrics exceeding the baseline performance metrics, provide an indication that the code change improves the model training pipeline with respect to at least one defect type of the plurality of defect types, whereby a baseline version of the model training pipeline is updated to include the code change and the baseline performance metrics are updated based on the test performance metrics.

7. The system of claim 1 , wherein the memory and the computer program code are configured to, with the processor, further cause the processor to:
provide an indicator of the identified at least one defect, wherein the indicator includes the defect type of the defect of the indicator and the lifecycle stage with which the defect of the indicator is associated, whereby a defect correction process is enabled to remedy the at least one defect based on the associated defect type and lifecycle stage.

8. The system of claim 1 , wherein the test performance metrics are associated with each defect type of the plurality of defect types based on each corresponding test model of the plurality of test models.

9. A computerized method for managing a defect in a model training pipeline, the computerized method comprising:
generating, by a processor, a synthetic data set, wherein the synthetic data set is associated with a defect type and a lifecycle stage in the model training pipeline;
generating, by the processor, baseline performance metrics based on the generated synthetic data set, wherein the baseline performance metrics are associated with the defect type;
based on a code change being made to the model training pipeline, training a test model, by the processor, using the model training pipeline and the generated synthetic data set; and
based on comparing the baseline performance metrics to test performance metrics, identifying, by the processor, a defect of the defect type in the model training pipeline.

10. The computerized method of claim 9 , wherein generating the baseline performance metrics based on the generated synthetic data set includes:
training a baseline model, by the processor, using a baseline version of the model training pipeline and the generated synthetic data set; and
based on the training of the baseline model using the model training pipeline, collecting, by the processor, the baseline performance metrics using on the baseline model, wherein the collected baseline performance metrics are associated with the defect type and lifecycle stage of the generated synthetic data set.

11. The computerized method of claim 10 , wherein the baseline performance metrics include a baseline model accuracy percentage and the test performance metrics include a test model accuracy percentage;
wherein comparing the baseline performance metrics and the test performance metrics includes comparing a difference between the baseline model accuracy percentage and the test model accuracy percentage to a threshold value; and
wherein identifying the defect of the defect type in the model training pipeline is further based on the difference between the baseline model accuracy percentage and the test model accuracy percentage exceeding the threshold value and the test model accuracy percentage being less than the baseline model accuracy percentage.

12. The computerized method of claim 9 , wherein generating the synthetic data set includes transforming a real data set to include a data pattern associated with the defect type, whereby a defect of the defect type in the model training pipeline causes performance of models trained using the model training pipeline and the transformed real data set with the data pattern to degrade from the baseline performance me

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Wang Shoou-Jiun
• Zhang Xing
• Abdelreheem Eslam K

Classifications

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title