User acceptance test system for machine learning systems

Patent metadata

Field	Value
Publication number	US-2022300754-A1
Application number	US-202117203921-A
Country	US
Kind code	A1
Filing date	Mar 17, 2021
Priority date	Mar 17, 2021
Publication date	Sep 22, 2022
Grant date	—

Abstract

Methods, systems, and computer-readable storage media for receiving, by a ML application executing within a cloud platform, a first inference request, the first inference request including first inference data, transmitting, by the ML application, the first inference data to the UAT system within the cloud platform, retrieving, by the UAT system, a first ML model in response to the inference request, the first ML model being in an inactive state, providing, by the UAT system, a first inference based on the first inference data using the first ML model, providing a first accuracy evaluation at least partially based on the first inference, and transitioning the first ML model from the inactive state to an active state, the first ML model being used for production in the active state.

First claim

What is claimed is:

1 . A computer-implemented method for selectively deploying machine learning (ML) models to production using a user acceptance test (UAT) system, the method comprising:
receiving, by a ML application executing within a cloud platform, a first inference request, the first inference request comprising first inference data; transmitting, by the ML application, the first inference data to the UAT system within the cloud platform; retrieving, by the UAT system, a first ML model in response to the inference request, the first ML model being in an inactive state; providing, by the UAT system, a first inference based on the first inference data using the first ML model; providing a first accuracy evaluation at least partially based on the first inference; and transitioning the first ML model from the inactive state to an active state, the first ML model being used for production in the active state.

2 . The method of claim 1 , further comprising:
generating, by the ML application, a second inference based on the first inference data using a second ML model in parallel with generating the first inference, the second ML model being in the active state; and replacing the second ML model with the first ML model for subsequent production use in response to transitioning the first ML model to the active state.

3 . The method of claim 2 , wherein the first ML model is an updated version of the second ML model.

4 . The method of claim 1 , wherein the first accuracy evaluation comprises:
determining an accuracy of the first ML model that represents correct inferences of the first ML model; and comparing the accuracy of the first ML model to a threshold accuracy.

5 . The method of claim 1 , wherein providing a first accuracy evaluation is executed in response to occurrence of a polling condition.

6 . The method of claim 1 , wherein the first inference data comprises production data.

7 . The method of claim 1 , further comprising:
retrieving, by the UAT system, a second ML model in response to a second inference request, the second ML model being in an inactive state; providing, by the UAT system, a second inference based on second inference data of the second inference request using the second ML model; determining a second accuracy evaluation at least partially based on the second inference; and transmitting an alert regarding the second ML model in response to the second accuracy evaluation.

8 . A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for selectively deploying machine learning (ML) models to production using a user acceptance test (UAT) system, the operations comprising:
receiving, by a ML application executing within a cloud platform, a first inference request, the first inference request comprising first inference data; transmitting, by the ML application, the first inference data to the UAT system within the cloud platform; retrieving, by the UAT system, a first ML model in response to the inference request, the first ML model being in an inactive state; providing, by the UAT system, a first inference based on the first inference data using the first ML model; providing a first accuracy evaluation at least partially based on the first inference; and transitioning the first ML model from the inactive state to an active state, the first ML model being used for production in the active state.

9 . The non-transitory computer-readable storage medium of claim 8 , wherein operations further comprise:
generating, by the ML application, a second inference based on the first inference data using a second ML model in parallel with generating the first inference, the second ML model being in the active state; and replacing the second ML model with the first ML model for subsequent production use in response to transitioning the first ML model to the active state.

10 . The non-transitory computer-readable storage medium of claim 9 , wherein the first ML model is an updated version of the second ML model.

11 . The non-transitory computer-readable storage medium of claim 8 , wherein the first accuracy evaluation comprises:
determining an accuracy of the first ML model that represents correct inferences of the first ML model; and comparing the accuracy of the first ML model to a threshold accuracy.

12 . The non-transitory computer-readable storage medium of claim 8 , wherein providing a first accuracy evaluation is executed in response to occurrence of a polling condition.

13 . The non-transitory computer-readable storage medium of claim 8 , wherein the first inference data comprises production data.

14 . The non-transitory computer-readable storage medium of claim 8 , wherein operations further comprise:
retrieving, by the UAT system, a second ML model in response to a second inference request, the second ML model being in an inactive state; providing, by the UAT system, a second inference based on second inference data of the second inference request using the second ML model; determining a second accuracy evaluation at least partially based on the second inference; and transmitting an alert regarding the second ML model in response to the second accuracy evaluation.

15 . A system, comprising:
a computing device; and a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for selectively deploying machine learning (ML) models to production using a user acceptance test (UAT) system, the operations comprising:
receiving, by a ML application executing within a cloud platform, a first inference request, the first inference request comprising first inference data;
transmitting, by the ML application, the first inference data to the UAT system within the cloud platform;
retrieving, by the UAT system, a first ML model in response to the inference request, the first ML model being in an inactive state;
providing, by the UAT system, a first inference based on the first inference data using the first ML model;
providing a first accuracy evaluation at least partially based on the first inference; and
transitioning the first ML model from the inactive state to an active state, the first ML model being used for production in the active state.

16 . The system of claim 15 , wherein operations further comprise:
generating, by the ML application, a second inference based on the first inference data using a second ML model in parallel with generating the first inference, the second ML model being in the active state; and replacing the second ML model with the first ML model for subsequent production use in response to transitioning the first ML model to the active state.

17 . The system of claim 16 , wherein the first ML model is an updated version of the second ML model.

18 . The system of claim 15 , wherein the first accuracy evaluation comprises:
determining an accuracy of the first ML model that represents correct inferences of the first ML model; and comparing the accuracy of the first ML model to a threshold accuracy.

19 . The system of claim 15 , wherein providing a first accuracy evaluation is executed in response to occurrence of a polling condition.

20 . The system of claim 15 , wherein the first infer

Assignees

• Sap Se

Inventors

• Biswas Atreya
• Gee Denny Jee King
• Santhanam Srivatsan

Classifications

• G06F18/2178

  based on feedback of a supervisor · CPC title

• G06F18/285 Primary

  Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title