Multi-client service system platform

Patent metadata

Field	Value
Publication number	US-2020210867-A1
Application number	US-201916716688-A
Country	US
Kind code	A1
Filing date	Dec 17, 2019
Priority date	Dec 27, 2018
Publication date	Jul 2, 2020
Grant date	—

Abstract

A modular machine learning-as-a-service (MLAAS) system uses machine learning to respond to tasks without requiring machine learning modeling or design knowledge by its users. The MLAAS system receives an inference request including a model identifier and a target defining features for use in processing the inference request. The features correspond to a task for evaluation using a machine learning model associated with the model identifier. An inference outcome is generated by processing the inference request using the target as input to the model. Feedback indicating an accuracy of the inference outcome with respect to the task is later received and used to generate a training data set, which the MLAAS can use to further train model used to generate the inference outcome. As a result, the training of a machine learning model by the MLAAS system is limited to using data resulting from an inference performed using that model.

First claim

What is claimed is:

1 . A machine learning-as-a-service (MLAAS) system, comprising:
a data store that stores machine learning models; and a cognitive processes system structured to process an inference request and train one or more machine learning models by: receiving, from a computer, the inference request including a model identifier and a target defining a set of features for use in processing the inference request, wherein the set of features correspond to a task for evaluation using the one or more machine learning models, from the data store, associated with the model identifier; determining based on the model identifier one or more additional features for processing the inference request, wherein the one or more additional features are related to at least one of the features in the set of features; retrieving the one or more additional features from one or more data sources; generating an inference outcome by processing the inference request using the target and the one or more additional features as input to the one or more machine learning models; transmitting the inference outcome to the computer; and training the one or more machine learning models based on feedback resulting from the inference outcome, the feedback indicating one of an accuracy of the inference outcome with respect to the task received by the cognitive processes system and a default feedback based on the task.

2 . The MLAAS system of claim 1 further comprising: a training data data store that stores a training data set used by the cognitive processes system to train the one or more machine learning models, wherein the data store is a model data store, and wherein a training data set corresponding to the inference request includes the feedback.

3 . The MLAAS system of claim 2 , wherein the training data set further includes an indication of a version of the one or more machine learning models used to generate the inference outcome, wherein training the one or more machine learning models based on the feedback comprises:
training the version of the one or more machine learning models based on the feedback.

4 . The MLAAS system of claim 1 , wherein generating the inference outcome by processing the inference request using the target as input to the one or more machine learning models comprises:
generating a plurality of candidate inference outcomes based on different processing of the target using the one or more machine learning models; and using a selection strategy to select one of the plurality of candidate inference outcomes as the inference outcome.

5 . The MLAAS system of claim 4 , wherein the selection strategy is a maximum likelihood estimation, wherein the cognitive processes system uses the maximum likelihood estimation to select the inference outcome based on the inference outcome having a highest score amongst the plurality of candidate inference outcomes.

6 . The MLAAS system of claim 4 , wherein the selection strategy is a multi-arm bandit approach, wherein the cognitive processes system uses the multi-arm bandit approach to select the inference outcome based on a distribution of values of each of the plurality of candidate inference outcomes after a number of inference iterations are completed.

7 . The MLAAS system of claim 1 , wherein the inference request is included in an application programming interface (API) call, wherein the API call includes a request identifier for the inference request, the model identifier, and the target.

8 . The MLAAS system of claim 7 , wherein the API call is a first API call, wherein the feedback is included in a second API call, wherein the second API call includes the request identifier and an outcome indicating the accuracy of the inference outcome with respect to the task.

9 . The MLAAS system of claim 8 , wherein training the one or more machine learning models based on the feedback comprises:
identifying a version of the one or more machine learning models used to generate the inference outcome based on the request identifier; and training the identified version of the one or more machine learning models using the outcome.

10 . The MLAAS system of claim 1 , wherein the computer is a computer used for a software product supported by the cognitive processes system.

11 . The MLAAS system of claim 10 , wherein at least one of the one or more machine learning models are received from the software product.

12 . The MLAAS system of claim 10 , wherein the software product and the cognitive processes system are both included in an application ecosystem.

13 . The MLAAS system of claim 1 , wherein the computer is a client associated with a software product supported by the cognitive processes system.

14 . The MLAAS system of claim 13 , wherein at least one of the one or more machine learning models is received from the client.

15 . The MLAAS system of claim 1 , wherein the machine learning models are generated using the cognitive processes system.

16 . The MLAAS system of claim 1 , wherein the cognitive processes system includes an inference system and a training system, wherein the inference system receives the inference request and generates the inference outcome, wherein the training system receives the feedback and trains the one or more machine learning models.

17 . A machine learning-as-a-service (MLAAS) system, comprising:
a data store that stores machine learning models; an enhancement system structured to enhance an inference request received from a computer, the inference request including a model identifier and a target defining a set of features for use in processing the inference request, the enhancement system configured to determine, based on the model identifier, one or more data sources of features from which the enhancement system retrieves one or more additional features based at least in part on the target, wherein the set of features corresponds to a task for evaluation using one or more machine learning models in the data store; an inference system structured to generate an inference outcome by processing the inference request using the target and the one or more additional features as input to one or more machine learning models that are retrieved from the data store using the model identifier and transmitting the inference outcome to the computer; and a machine learning model training system structured to train the retrieved one or more machine learning models based on feedback resulting from the inference outcome, the feedback comprising one of an accuracy of the inference outcome with respect to the task that is received by the enhancement system and a default feedback based on the task.

18 . The MLAAS system of claim 17 , wherein the data store is a model data store, the MLAAS system further comprising:
a training data data store that stores a training data set used by the enhancement system to train the one or more machine learning models, wherein a training data set corresponding to the inference request includes the feedback.

19 . The MLAAS system of claim 18 , wherein the training data set further includes an indication of a version of the one or more machine learning models used to generate the inference outcome, wherein training the one or more machine learning models based on the feedback comprises:
training the version of the one or more machine learning models based on the feedback.

20 . The MLAAS system of claim 17 , wherein generating the inference outcome by processing the inference request using the t

Assignees

• Hubspot Inc

Inventors

• Banis George
• Starikiewicz Adam
• Walsh Kevin M
• Purcell Stephen
• Urdiales Hector
• Bergonzo Andrea

Classifications

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title