Multi-client service system platform

Patent metadata

Field	Value
Publication number	US-11727287-B2
Application number	US-202217882950-A
Country	US
Kind code	B2
Filing date	Aug 8, 2022
Priority date	Dec 27, 2018
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

A modular machine learning-as-a-service (MLAAS) system uses machine learning to respond to tasks without requiring machine learning modeling or design knowledge by its users. The MLAAS system receives an inference request including a model identifier and a target defining features for use in processing the inference request. The features correspond to a task for evaluation using a machine learning model associated with the model identifier. An inference outcome is generated by processing the inference request using the target as input to the model. Feedback indicating an accuracy of the inference outcome with respect to the task is later received and used to generate a training data set, which the MLAAS can use to further train model used to generate the inference outcome. As a result, the training of a machine learning model by the MLAAS system is limited to using data resulting from an inference performed using that model.

First claim

What is claimed is:

1. A method, comprising:
receiving an inference request including a model identifier and a target defining a set of features for use in processing the inference request, wherein the set of features correspond to a task for evaluation using a machine learning model associated with the model identifier;
determining based on the model identifier an additional feature, related to the set of features, for processing the inference request;
generating an inference outcome corresponding to the inference request being processed using the target and the additional feature as input to the machine learning model; and
training the machine learning model based on feedback resulting from the inference outcome, the feedback indicating an accuracy of the inference outcome with respect to the task.

2. The method of claim 1 , wherein the feedback indicates a default feedback based on the task.

3. The method of claim 1 , wherein the training comprises:
generating a training data set based upon the inference request and the feedback, wherein the training data set is used to train the machine learning model.

4. The method of claim 1 , wherein the inference request is included in an application programming interface (API) call, wherein the API call includes a request identifier for the inference request, the model identifier, and the target.

5. The method of claim 1 , comprising:
using a selection strategy to select one of a plurality of candidate inference outcomes, generated based on different processing of the target using the machine learning model, as the inference outcome.

6. The method of claim 1 , comprising:
using a selection strategy to select one of a plurality of candidate inference outcomes, generated based on different processing of the target using the machine learning model, as the inference outcome, wherein the selection strategy is a maximum likelihood estimation used to select the inference outcome based on the inference outcome having a highest score amongst the plurality of candidate inference outcomes.

7. The method of claim 1 , comprising:
using a selection strategy to select one of a plurality of candidate inference outcomes, generated based on different processing of the target using the machine learning model, as the inference outcome, wherein the selection strategy is a multi-arm bandit approach used to select the inference outcome based on a distribution of values of each of the plurality of candidate inference outcomes after a number of inference iterations are completed.

8. The method of claim 1 , comprising:
limiting the training of the machine learning model to data from an inference perform using the machine learning model based upon the inference request.

9. The method of claim 1 , comprising:
selectively training a version of the machine learning model, used to generate the inference outcome based on a request identifier, using the inference outcome.

10. A non-transitory machine readable medium comprising instructions, which when executed by a machine, causes the machine to perform operations including:
receiving an inference request including a model identifier and a target defining a set of features for use in processing the inference request, wherein the set of features correspond to a task for evaluation using a machine learning model associated with the model identifier;
determining based on the model identifier an additional feature, related to the set of features, for processing the inference request;
generating an inference outcome corresponding to the inference request being processed using the target and the additional feature as input to the machine learning model; and
training the machine learning model based on feedback resulting from the inference outcome, the feedback indicating a default feedback based on the task.

11. The non-transitory machine readable medium of claim 10 , wherein the feedback indicates an accuracy of the inference outcome with respect to the task.

12. The non-transitory machine readable medium of claim 10 , the operations including:
generating a training data set based upon the inference request and the feedback, wherein the training data set is used to train the machine learning model.

13. The non-transitory machine readable medium of claim 10 , wherein the inference request is included in an application programming interface (API) call, wherein the API call includes a request identifier for the inference request, the model identifier, and the target.

14. The non-transitory machine readable medium of claim 10 , the operations including:
using a selection strategy to select one of a plurality of candidate inference outcomes, generated based on different processing of the target using the machine learning model, as the inference outcome.

15. The non-transitory machine readable medium of claim 10 , the operations including:
using a selection strategy to select one of a plurality of candidate inference outcomes, generated based on different processing of the target using the machine learning model, as the inference outcome, wherein the selection strategy is a maximum likelihood estimation used to select the inference outcome based on the inference outcome having a highest score amongst the plurality of candidate inference outcomes.

16. The non-transitory machine readable medium of claim 10 , the operations including:
using a selection strategy to select one of a plurality of candidate inference outcomes, generated based on different processing of the target using the machine learning model, as the inference outcome, wherein the selection strategy is a multi-arm bandit approach used to select the inference outcome based on a distribution of values of each of the plurality of candidate inference outcomes after a number of inference iterations are completed.

17. The non-transitory machine readable medium of claim 10 , the operations including:
limiting the training of the machine learning model to data from an inference perform using the machine learning model based upon the inference request.

18. A computing device comprising:
a memory comprising machine executable code for performing a method; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to perform operations including:
receiving an inference request including a model identifier and a target defining a set of features for use in processing the inference request, wherein the set of features correspond to a task for evaluation using a machine learning model associated with the model identifier;
determining based on the model identifier an additional feature, related to the set of features, for processing the inference request;
generating an inference outcome corresponding to the inference request being processed using the target and the additional feature as input to the machine learning model; and
training the machine learning model based on feedback resulting from the inference outcome, the feedback indicating an accuracy of the inference outcome with respect to the task.

19. The computing device of claim 18 , the operations including:
using a selection strategy to select one of a plurality of candidate inference outcomes, generated based on different processing of the target using the machine learning model, as the inference outcome, wherein the selection strategy is a maximum likelihood estimation used to select the inference outcome based on the inference outcome having a highest score amongst the pluralit

Assignees

• Hubspot Inc

Inventors

• Banis George
• Starikiewicz Adam
• Walsh Kevin M
• Purcell Stephen
• Urdiales Hector
• Bergonzo Andrea

Classifications

• G06N5/04 Primary

  Inference or reasoning models · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title