Generating reliability measures for machine-learned architecture predictions

Patent metadata

Field	Value
Publication number	US-12051008-B2
Application number	US-202217883503-A
Country	US
Kind code	B2
Filing date	Aug 8, 2022
Priority date	Aug 8, 2022
Publication date	Jul 30, 2024
Grant date	Jul 30, 2024

Abstract

A prediction system of an online system deploys one or more machine-learned architectures to generate predictions. In one embodiment, the machine-learned architecture is a stacked ensemble model. The stacked ensemble model includes a plurality of base models, where a base model is coupled to receive input data and generate a base prediction for the input data. The stacked ensemble model includes a meta model that combines the base predictions to generate a meta prediction for the input data. The prediction system also generates a reliability measure that takes advantage of the base predictions to evaluate the reliability of the meta prediction. In this manner, while the quality of individual predictions may differ from one another depending on the values of the input data, the prediction system can dynamically generate the reliability measure to account for this variation.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
training, at a computing device, a machine-learned model of a plurality of machine-learned models stored in a database communicatively coupled to the computing device by using a forward pass and backpropagation to reduce a loss function, wherein the backpropagation updates parameters based on error terms obtained from the loss function, which is repeated until a convergence criterion is reached for a set of parameters;
accessing, at the computing device from the database, the machine-learned model coupled to receive input data and generate a base prediction indicating an estimated value for a target variable;
responsive to receiving a request including a query received by the computing device:
generating, at the computing device, a plurality of base predictions for the query by applying the plurality of machine-learned models to the query, and
generating, at the computing device a meta prediction by combining the base predictions for the query;

generating, at the computing device, a posterior distribution for the target variable given the meta prediction, the generating comprising:
obtaining a prior distribution of the target variable,
generating a likelihood function of the meta prediction given values of the target variable, wherein the likelihood function is determined based on the meta prediction and the plurality of base predictions, and
generating the posterior distribution of the target variable given the meta prediction by combining the prior distribution and the likelihood function;

determining, at the computing device, a reliability measure with respect to a confidence level for the meta prediction that indicates a range the meta prediction could be in based on the posterior distribution; and
providing, at the computing device, at least the meta prediction and the reliability measure as a response to the request.

2. The computer-implemented method of claim 1 , wherein the posterior distribution is generated by at least multiplying the prior distribution of the target variable with the likelihood function.

3. The computer-implemented method of claim 1 , wherein obtaining the prior distribution further comprises:
identifying a plurality of data instances for the target variable, a data instance including input data and a known value of the target variable for the data instance; and
obtaining the prior distribution of the target variable from a frequency histogram of the known values of the target variable for the plurality of data instances or a distribution fitted to the known values of the target variable for the plurality of data instances.

4. The computer-implemented method of claim 1 , wherein for a given value of the target variable, the likelihood function for the meta prediction is a Gaussian distribution having a mean at the value of the target variable and a meta prediction variance.

5. The computer-implemented method of claim 4 , wherein a base prediction is associated with a respective weight, and the meta prediction is generated by:
weighting each base prediction with the respective weight for the base prediction, and combining the weighted base predictions to generate the meta prediction.

6. The computer-implemented method of claim 5 , wherein the meta prediction variance in the likelihood function is estimated by:
determining, for each base prediction, a deviation of the base prediction from the value of the target variable and weighting the deviation for the base prediction with the respective weight for the base prediction, and
combining the weighted deviations to generate the estimate for the meta prediction variance.

7. The computer-implemented method of claim 6 , wherein the value of the target variable in the estimate for the meta prediction variance is estimated as a weighted arithmetic mean of the base predictions or a weighted average of the base predictions.

8. The computer-implemented method of claim 1 , further comprising:
responsive to receiving a second request including a second query:
generating a second plurality of base predictions for the second query by applying the plurality of machine-learned models to the second query, and
generating a second meta prediction by combining the base predictions for the second query;

generating a second reliability measure for the second meta prediction, comprising:
generating a second likelihood function of the second meta prediction based on a value of the second meta prediction and the second plurality of base predictions, and
generating a second posterior distribution of the target variable given the second meta prediction by combining the prior distribution and the second likelihood function, wherein the second reliability measure is different from the reliability measure.

9. The computer-implemented method of claim 1 , wherein a first machine-learned model of the plurality of machine-learned models is of a different type of model, a different structure, or is associated with a different set of parameters than a second machine-learned model of the plurality of machine-learned models.

10. The computer-implemented method of claim 1 , wherein the query includes information describing a user of a client device and the target variable is a likelihood the user will open a marketing message sent to the user.

11. A computer-implemented method, comprising:
training, at a computing device, a machine-learned model of a plurality of machine-learned models stored in a database communicatively coupled to the computing device by using a forward pass and backpropagation to reduce a loss function, wherein the backpropagation updates parameters based on error terms obtained from the loss function, which is repeated until a convergence criterion is reached for a set of parameters;
responsive to receiving a request to generate one or more predictions for a target variable, generating, at the computing device, a plurality of base predictions for a query of the request by applying the plurality of machine-learned models to the query and generating a meta prediction by combining the base predictions for the query;
generating, at the computing device, a posterior distribution for the target variable given the meta prediction, the generating comprising:
obtaining a prior distribution of the target variable,
generating a likelihood function of the meta prediction given values of the target variable, wherein the likelihood function is determined based on the meta prediction and the plurality of base predictions, and
generating the posterior distribution of the target variable given the meta prediction by combining the prior distribution and the likelihood function;

determining, at the computing device, the reliability measure for the meta prediction that indicates a degree of confidence in the meta prediction based on the posterior distribution; and
providing, at the computing device, at least the meta prediction and the reliability measure as a response to the request.

12. The computer-implemented method of claim 11 , wherein the posterior distribution is generated by at least multiplying the prior distribution of the target variable with the likelihood function.

13. The computer-implemented method of claim 11 , wherein obtaining the prior distribution further comprises:
identifying a plurality of data instances for the target variable, a data instance including input data and a known value of the target variable for the data instance; and
obtaining the prior distribution of the target variable from a frequency histogram of the known values of the target variable for the plurality of data instance

Assignees

• Salesforce Inc

Inventors

• Hu Donglin
• Zhang Yuxi
• Xie Kexin
• Xu Chen

Classifications

• G06Q30/0241 Primary

  Advertisements · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N20/00

  Machine learning · CPC title