Machine learning platform for dynamic model selection

Patent metadata

Field	Value
Publication number	US-11720813-B2
Application number	US-201816147255-A
Country	US
Kind code	B2
Filing date	Sep 28, 2018
Priority date	Sep 29, 2017
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

The present disclosure relates generally to an integrated machine learning platform. The machine learning platform can convert machine learning models with different schemas into machine learning models that share a common schema, organize the machine learning models into model groups based on certain criteria, and perform pre-deployment evaluation of the machine learning models. The machine learning models in a model group can be evaluated or used individually or as a group. The machine learning platform can be used to deploy a model group and a selector in a production environment, and the selector may learn to dynamically select the model(s) from the model group in the production environment in different contexts or for different input data, based on a score determined using certain scoring metrics, such as certain business goals.

First claim

What is claimed is:

1. A method comprising, by a computing system:
selecting one or more machine learning (ML) models from a plurality of ML models;
determining a common schema for a model group that includes the one or more ML models, wherein the common schema indicates names and datatypes of features of the one or more ML models and each ML model in the model group is configured to perform a same function;
converting a first ML model having a schema different from the common schema to a converted first ML model having the common schema;
adding the converted first ML model to the model group;
dynamically selecting, by a model selector for the model group and based on a set of rules or a trainable selection model, at least one ML model from the model group for data analysis;
analyzing input data using the model group including the at least one dynamically selected ML model, and the model selector;
determining, during the analyzing, a score for the model group including the at least one dynamically selected ML model, and the model selector based on the analyzing and a set of scoring metrics; and
updating, during the analyzing, the model selector or the model group based upon determining that the score is below a threshold value.

2. The method of claim 1 , wherein a feature in the first ML model has a same name as and a different datatype than a corresponding feature in the common schema.

3. The method of claim 1 , wherein the one or more ML models in the model group include different versions of a machine learning model.

4. The method of claim 1 , wherein the set of scoring metrics comprises a business goal.

5. The method of claim 1 , wherein the set of rules includes a rule for selecting the at least one ML model based on attributes of the input data.

6. The method of claim 1 , wherein updating the model selector comprises:
adding a new rule to the set of rules;
revising a rule in the set of rules; or
revising the trainable selection model.

7. The method of claim 1 , wherein updating the model group comprises:
retraining the first ML model in the model group based on the analyzing and the score; and
adding the retrained first ML model to the model group.

8. The method of claim 1 , wherein the input data includes real-time input data from a production environment.

9. The method of claim 8 , wherein the input data includes contextual data of the production environment.

10. The method of claim 1 , wherein analyzing the input data using the model group and the model selector comprises:
analyzing a first portion of the input data using the first ML model in the model group; and
analyzing a second portion of the input data using a second ML model in the model group.

11. The method of claim 10 , wherein analyzing the input data using the model group and the model selector further comprises analyzing a third portion of the input data using a third ML model in the model group.

12. The method of claim 1 , wherein the model selector is further configured to determine a scheme for using the selected at least one ML model to analyze the input data.

13. The method of claim 12 , wherein the scheme for using the selected at least one ML model to analyze the input data comprises:
for each of the selected at least one ML model, analyzing a same portion of the input data by the ML model to obtain a corresponding result; and
selecting, as a result for the portion of the input data, a most common result from the corresponding results.

14. The method of claim 1 , further comprising:
reporting usage of the one or more ML models in the model group for the analyzing.

15. The method of claim 1 ,
wherein converting the first ML model comprises converting a datatype of a feature in the first ML model.

16. The method of claim 1 , wherein determining the common schema for the one or more ML models comprises:
adding one of two congruent features in two respective schemas for two ML models to the common schema; or
dropping a feature in a schema for a second ML model based on determining that the feature has an importance level below a second threshold value.

17. A non-transitory computer readable medium storing a plurality of instructions executable by one or more processors, wherein the plurality of instructions, when executed by the one or more processors, causes the one or more processors to perform processing comprising:
selecting one or more machine learning (ML) models from a plurality of ML models;
determining a common schema for a model group that includes the one or more ML models, wherein the common schema indicates names and datatypes of features of the one or more ML models and each ML model in the model group is configured to perform a same function;
converting a first ML model having a schema different from the common schema to a converted first ML model having the common schema;
adding the converted first ML model to the model group;
dynamically selecting, by a model selector for the model group and based on a set of rules or a trainable selection model, at least one ML model from the model group for data analysis;
analyzing input data using the model group including the at least one dynamically selected ML model, and the model selector;
determining, during the analyzing, a score for the model group including the at least one dynamically selected ML model, and the model selector based on the analyzing and a set of scoring metrics; and
updating, during the analyzing, the model selector or the model group based upon determining that the score is below a threshold value.

18. The non-transitory computer readable medium of claim 17 , wherein determining the common schema for the model group comprises:
determining the common schema based on a union of schemas for the one or more ML models;
adding one of two congruent features in two respective schemas for two ML models to the common schema; or
dropping a feature in a schema for a second ML model based on determining that the feature has an importance level below a second threshold value.

19. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory storing instructions, which, when executed by the one or more processors, cause the system to:
select one or more machine learning (ML) models from a plurality of ML models;
determine a common schema for a model group that includes the one or more ML models, wherein the common schema indicates names and datatypes of features of the one or more ML models and each ML model in the model group is configured to perform a same function;
convert a first ML model having a schema different from the common schema to a converted first ML model having the common schema;
add the converted first ML model to the model group;
dynamically select, by a model selector for the model group and based on a set of rules or a trainable selection model, at least one ML model from the model group for data analysis;
analyze input data using the model group including the at least one dynamically selected ML model, and the model selector;
determine, during the analyzing, a score for the model group including the at least one dynamically selected ML model, and the model selector based on the analyzing and a set of scoring metrics; and
update, during the analyzing, the model selector or the model group based upon determining that the score is below a threshold value.

20. The method according to claim 1 , wherein the first ML model having the schema different fro

Assignees

• Oracle Int Corp

Inventors

• Babu Shashi Anand
• Venkatasubramanian Raghuram
• Madhav Neel
• Mazoyer Herve
• Race Daren
• Sundaram Arun Kumar Kalyaana
• Thilagar Lasya Priya

Classifications

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/046

  Forward inferencing; Production systems · CPC title

• G06N20/20 Primary

  Ensemble learning · CPC title

• G06N5/025

  Extracting rules from data · CPC title