Early stopping of artificial intelligence model training using control limits

Patent metadata

Field	Value
Publication number	US-12468979-B2
Application number	US-202117460689-A
Country	US
Kind code	B2
Filing date	Aug 30, 2021
Priority date	Aug 30, 2021
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

Disclosed herein is a method of training an artificial intelligence model that comprises an iterative training loop. Said iterative training loop comprises: receiving a current set of training data; dividing said current set of training data into a predetermined number of training data subsets; sequentially training said artificial intelligence model with each of said predetermined number of training data subsets using a training portion and calculating a performance metric using a validation portion; and comparing performance metrics from a previous iteration of said iterative training loop to said calculated performance metric to determine if an improving performance metric condition is met. Said method further comprises halting said iterative training loop unless said improving performance metric condition is not met at least once within a predetermined number of previous iterations.

First claim

What is claimed is:

1 . A method of training an artificial intelligence model, said method comprising an iterative training loop, wherein said iterative training loop comprises:
receiving a current set of training data, wherein said current set of training data is different from training data from previous iterations of said iterative training loop; dividing said current set of training data into a plurality of training data subsets, each training data subset having a respective training portion and a respective validation portion; sequentially training an artificial intelligence model with said training data subsets using said respective training portions, wherein sequentially training said artificial intelligence model includes calculating a performance metric for each training data subset of said training data subsets using said respective validation portion after training using said respective training portion; comparing performance metrics from a previous iteration of said iterative training loop to said calculated performance metrics for said training data subsets to determine whether an improving performance condition is met; and halting said iterative training loop responsive to determining the improving performance metric condition is met at least once within a predetermined number of previous iterations.

2 . The method of claim 1 , further comprising training multiple artificial intelligence models simultaneously using said iterative training loop.

3 . The method of claim 1 , wherein the comparing performance metrics comprises comparing a control limit of a previous iteration of said iterative training loop to a calculated current control limit, wherein the control limits are calculated based on a mean of said performance metrics during a respective iteration and a range of said performance metrics during said respective iteration.

4 . The method of claim 1 , wherein said performance metric is selected from the group consisting of: accuracy, precision, recall, area under curve metric, true positive rate, true negative rate, false positive rate, sum of said true positive rate and said true negative rate both divided by a total sample size, mean squared error, and F1 score.

5 . The method of claim 1 , wherein said artificial intelligence model is a convolutional neural network, and wherein said training of said artificial intelligence model is performed using a deep learning algorithm.

6 . The method of claim 1 , wherein said artificial intelligence model is selected from the group consisting of: a neural network, a classifier neural network, a convolutional neural network, a Bayesian neural network, a Bayesian network, a Bayes network, naive Bayes classifiers, belief network, or decision network, a decision trees, a support-vector machine, a regression analysis, and a genetic algorithm.

7 . The method of claim 2 , wherein each of said multiple artificial intelligence models is selected from the group consisting of: a neural network, a classifier neural network, a convolutional neural network, a Bayesian neural network, a Bayesian network, a Bayes network, naive Bayes classifiers, belief network, or decision network, a decision trees, a support-vector machine, a regression analysis, and a genetic algorithm.

8 . A computer system comprising:
a processor configured for controlling the computer system; and a memory storing machine executable instructions, wherein execution of said instructions causes said processor to perform an iterative training loop, wherein said iterative training loop comprises:
receiving a current set of training data, wherein said current set of training data is different from training data from previous iterations of said iterative training loop;
dividing said current set of training data into a plurality of training data subsets, each training data subset having a respective training portion and a respective validation portion;
sequentially training an artificial intelligence model with said training data subsets using said respective training portions, wherein sequentially training said artificial intelligence model includes calculating a performance metric for each training data subset of said training data subsets using said respective validation portion after training using said respective training portion;
comparing performance metrics from a previous iteration of said iterative training loop to said calculated performance metric for said training data subsets to determine whether an improving performance condition is met; and
halting said iterative training loop responsive to determining the improving performance metric condition is met at least once within a predetermined number of previous iterations.

9 . The computer system of claim 8 , wherein execution of said instructions further causes said processor to train multiple artificial intelligence models simultaneously using said iterative training loop.

10 . The computer system of claim 8 , wherein the comparing performance metrics comprises comparing a control limit of a previous iteration of said iterative training loop to a calculated current control limit, wherein the control limits are calculated based on a mean of said performance metrics during a respective iteration and a range of said performance metrics during said respective iteration.

11 . A method of training an artificial intelligence model, said method comprising an iterative training loop, wherein said iterative training loop comprises:
receiving a current set of training data, wherein said current set of training data is different from training data from previous iterations of said iterative training loop; dividing said current set of training data into a plurality of training data subsets, each training data subset having a respective training portion and a respective validation portion; sequentially training an artificial intelligence model with said training data subsets using said respective training portions, wherein sequentially training said artificial intelligence model includes calculating a performance metric for each training data subset of said training data subsets using said respective validation portion after training using said respective training portion; calculating a current training control limit based, at least in part, on the calculated performance metrics for said training data subsets; comparing a training control limit from a previous iteration of said iterative training loop to the calculated current training control limit to determine whether an improving performance metric condition is met; and halting said iterative training loop responsive to determining the improving performance metric condition is met at least once within a predetermined number of previous iterations.

12 . The method of claim 11 , wherein said method further comprises training multiple artificial intelligence models simultaneously using said iterative training loop.

13 . The method of claim 12 , wherein said method further comprises choosing a global control limit for each iteration of said iterative training loop, wherein said global control limit is selected by choosing said control limit from said iterative training loop that indicates a best performance metric of said multiple artificial intelligence models, and wherein said iterative training loop further comprises replacing said training control limit from an immediately previous iteration of said iterative training loop for each of said multiple artificial intelligence models with an immediately previous global control limit from an immediately previous iteration of said iterative training loop.

Assignees

• IBM

Inventors

• Cmielowski Lukasz G
• Ryszka Daniel Jakub
• Sobala Wojciech
• Bramble Gregory

Classifications

• G06N20/00 Primary

  Machine learning · CPC title