Techniques for adaptive pipelining composition for machine learning (ML)

Patent metadata

Field	Value
Publication number	US-12118474-B2
Application number	US-202318132859-A
Country	US
Kind code	B2
Filing date	Apr 10, 2023
Priority date	Sep 14, 2019
Publication date	Oct 15, 2024
Grant date	Oct 15, 2024

Abstract

The present disclosure relates to systems and methods for an adaptive pipelining composition service that can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.

First claim

What is claimed is:

1. A computer-implemented method, the method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics define how a performance of the first machine learning model is measured;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises a labeled data set;
comparing the first result with the second result to calculate a first scoring;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result;
generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model in a machine learning application.

2. The method of claim 1 , further comprising:
storing the second machine learning model in a memory.

3. The method of claim 1 , wherein the replacing the first machine learning model with the second machine learning model is performed in a shadow mode until it the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.

4. The method of claim 1 , further comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and
storing the log in a memory.

5. The method of claim 4 , further comprising analyzing the log to determine one or more patterns.

6. The method of claim 4 , further comprising saving supplemental metadata concerning the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the second result.

7. The method of claim 1 , wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.

8. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations executed as a background process of a machine learning application, the operations comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics determine a performance of the first machine learning model;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises ground truth data;
comparing the first result with the second result to calculate a first scoring;
analyzing one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result;
generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.

9. The computer-program product of claim 8 , including instructions configured to cause a data processing apparatus to perform further operations comprising storing the second machine learning model in a memory.

10. The computer-program product of claim 8 , wherein the replacing the first machine learning model is replaced by the second machine learning model in a shadow mode until the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.

11. The computer-program product of claim 8 , including instructions configured to cause a data processing apparatus to perform further operations comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and
storing the log in a memory.

12. The computer-program product of claim 11 , including instructions configured to cause a data processing apparatus to perform further operations comprising analyzing the log to determine one or more patterns.

13. The computer-program product of claim 11 , including instructions configured to cause a data processing apparatus to perform further operations comprising saving supplemental metadata concerning the second machine learning model based at least in part on the one or more first parameters the second result.

14. The computer-program product of claim 8 , wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.

15. A system for executed as a background process of a machine learning application, comprising:
one or more data processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics determine a performance of the first machine learning model;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises ground truth data;
comparing the first result to the second result to calculate a first scoring;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the sec

Assignees

• Oracle Int Corp

Inventors

• Polleri Alberto
• Suzuki Larissa Cristina Dos Santos Romualdo
• Lopez Sergio Aldea
• Bron Marc Michiel
• Golding Dan David
• Ioannides Alexander
• Mestre Maria Del Rosario
• Monteiro Hugo Alexandre Pereira
• Shevelev Oleg Gennadievich
• Zhao Xiaoxue
• Rowe Matthew Charles

Classifications

• G06N5/045

  Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence · CPC title

• G06F18/2193

  based on specific statistical tests · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/285

  Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system · CPC title

• G06F16/907

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title