System and method for training and refining machine learning models

Patent metadata

Field	Value
Publication number	US-2023072171-A1
Application number	US-202217837624-A
Country	US
Kind code	A1
Filing date	Jun 10, 2022
Priority date	Aug 30, 2021
Publication date	Mar 9, 2023
Grant date	—

Abstract

A system and method for training and refining a machine learning model is disclosed. The disclosed system and method can further improve the accuracy of trained machine learning models by calculating which threshold values for predictions (e.g., probabilities output by the machine learning model) provide the most accurate results. The system and method may include applying an optimization technique (e.g., multi-objective optimization) to calculate which threshold values result in the best combination of precision and recall. In other words, the system and method adjust threshold values for prediction scores to optimize the objects of precision and recall. A machine learning model trained with these adjusted threshold values can determine when an input belongs to an unknown class because the unknown input has prediction scores below the threshold values for every known class. Embodiments may include refining an intent classifier to better classify unknown intents.

First claim

We claim:

1 . A computer-implemented method of training and refining a machine learning model, comprising:
generating an initially trained machine learning model by training a machine learning model with a labeled dataset including labels defining which classes apply to each piece of data; inputting an unlabeled dataset with unknown classes into the initially trained machine learning model to generate prediction scores representing the probability that each piece of data belongs in each known class, respectively; randomly selecting multiple threshold values from a range of threshold values for prediction scores for each known class; for each known class, initializing a population with the randomly selected threshold values; calculating objective values of recall and precision for each of the randomly selected threshold values; performing multi-objective optimization to determine which threshold values of the randomly selected threshold values optimize both recall and precision for each known class; and generating a finally trained machine learning model by training the initially trained machine learning model to classify input using the determined optimal threshold values.

2 . The method of claim 1 , wherein performing multi-objective optimization comprises performing Non-dominated Sorting Genetic Algorithm II (NSGA-II) to determine which threshold values of the randomly selected threshold values optimize both recall and precision for each class the initially trained machine learning model is trained for.

3 . The method of claim 1 , wherein performing multi-objective optimization comprises sorting the initialized population into fronts ranked by ascending level of non-domination, wherein the fronts each contain members including threshold values of the randomly selected threshold values.

4 . The method of claim 3 , wherein performing multi-objective optimization comprises, for each front, calculating a crowding distance for each member of the respective front.

5 . The method of claim 4 , wherein performing multi-objective optimization comprises generating an offspring population by applying crowded tournament selection to the members of the fronts.

6 . The method of claim 5 , wherein crowded tournament selection includes comparing the rank between two members and, if one member has a higher rank than the other member, selecting the member with the higher rank, and, if two members have the same rank, selecting the member with the highest crowding distance.

7 . The method of claim 5 , wherein generating the offspring population includes applying crossover and mutation operators to the initialized population.

8 . The method of claim 1 , wherein the initially trained machine learning model is an intent classification deep neural network.

9 . A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to:
generate an initially trained machine learning model by training a machine learning model with a labeled dataset including labels defining which classes apply to each piece of data; input an unlabeled dataset with unknown classes into the initially trained machine learning model to generate prediction scores representing the probability that each piece of data belongs in each known class, respectively; randomly select multiple threshold values from a range of threshold values for prediction scores for each known class; for each known class, initialize a population with the randomly selected threshold values; calculate objective values of recall and precision for each of the randomly selected threshold values; perform multi-objective optimization to determine which threshold values of the randomly selected threshold values optimize both recall and precision for each known class; and generate a finally trained machine learning model by training the initially trained machine learning model to classify input using the determined optimal threshold values.

10 . The non-transitory computer-readable medium storing software of claim 9 , performing multi-objective optimization comprises performing Non-dominated Sorting Genetic Algorithm II (NSGA-II) to determine which threshold values of the randomly selected threshold values optimize both recall and precision for each class the initially trained machine learning model is trained for.

11 . The non-transitory computer-readable medium storing software of claim 9 , wherein performing multi-objective optimization comprises sorting the initialized population into fronts ranked by ascending level of non-domination, wherein the fronts each contain members including threshold values of the randomly selected threshold values.

12 . The non-transitory computer-readable medium storing software of claim 11 , wherein performing multi-objective optimization comprises, for each front, calculating a crowding distance for each member of the respective front.

13 . The non-transitory computer-readable medium storing software of claim 12 , wherein performing multi-objective optimization comprises generating an offspring population by applying crowded tournament selection to the members of the fronts.

14 . The non-transitory computer-readable medium storing software of claim 13 , wherein crowded tournament selection includes comparing the rank between two members and, if one member has a higher rank than the other member, selecting the member with the higher rank, and, if two members have the same rank, selecting the member with the highest crowding distance.

15 . The non-transitory computer-readable medium storing software of claim 14 , wherein generating the offspring population includes applying crossover and mutation operators to the initialized population.

16 . The non-transitory computer-readable medium storing software of claim 9 , wherein the initially trained machine learning model is an intent classification deep neural network.

17 . A system for training and refining a machine learning model, the system comprising:
a device processor; and a non-transitory computer readable medium storing instructions that are executable by the device processor to:
generate an initially trained machine learning model by training a machine learning model with a labeled dataset including labels defining which classes apply to each piece of data;
input an unlabeled dataset with unknown classes into the initially trained machine learning model to generate prediction scores representing the probability that each piece of data belongs in each known class, respectively;
randomly select multiple threshold values from a range of threshold values for prediction scores for each known class;
for each known class, initialize a population with the randomly selected threshold values;
calculate objective values of recall and precision for each of the randomly selected threshold values;
perform multi-objective optimization to determine which threshold values of the randomly selected threshold values optimize both recall and precision for each known class; and
generate a finally trained machine learning model by training the initially trained machine learning model to classify input using the determined optimal threshold values.

18 . The system of claim 17 , performing multi-objective optimization comprises performing Non-dominated Sorting Genetic Algorithm II (NSGA-II) to determine which threshold values of the randomly selected threshold values optimize both recall and precision for each class

Assignees

• Accenture Global Solutions Ltd

Inventors

• Sengupta Shubhashis
• Ramnani Roshni Ramesh
• Jain Sakshi
• Prem Prerna
• Ekbal Asif
• Ahmad Zishan

Classifications

• G10L15/063

  Training · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06F18/2113

  by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation · CPC title

• G06N3/086 Primary

  using evolutionary algorithms, e.g. genetic algorithms or genetic programming · CPC title