Systems and methods for evaluating performance of models

Patent metadata

Field	Value
Publication number	US-2020293424-A1
Application number	US-202016886806-A
Country	US
Kind code	A1
Filing date	May 29, 2020
Priority date	Nov 29, 2017
Publication date	Sep 17, 2020
Grant date	—

Abstract

A system and method for evaluating performance of difference models. The method may include: obtaining, by at least one computer, a first sample set and a second sample set; dividing, by the at least one computer, the first sample set into a plurality of first sample subsets, each first sample subset providing an average first sample subset characteristic value; dividing, by the at least one computer, the second sample set into a plurality of second sample subsets; each second sample subset providing an average second sample subset characteristic value; determining, by the at least one computer, a final model between the first model and the second model based on an average difference, a significance level, and a confidence interval between the first model and the second model.

First claim

1 . A system, comprising:
one or more storage media comprising a set of instructions for model evaluation; and one or more processors configured to communicate with the one or more storage media, wherein when executing the set of instructions, the one or more processors are directed to: (a) obtain a first sample set and a second sample set, wherein:
(i) the first sample set includes a plurality of first samples based on a first model,
(ii) the second sample set includes a plurality of second samples based on a second model, and
(iii) each of the first and second samples includes a characteristic value,
(b) divide the first sample set into a plurality of first sample subsets, each first sample subset providing an average first sample subset characteristic value; (c) divide the second sample set into a plurality of second sample subsets; each second sample subset providing an average second sample subset characteristic value; (d) determine a final model between the first model and the second model based on an average difference, a significance level, and a confidence interval between the first model and the second model, wherein the average difference, the significance level, and the confidence interval are based on the average first sample subset characteristic values and the average second sample subset characteristic values.

2 . The system of claim 1 , wherein to obtain the first sample set and the second sample set, for each sample, the one or more processors are further directed to:
obtain a request associated with a first randomizing parameter; assign the request to the first model or the second model based on the first randomizing parameter by using a first randomizing function; and generate the characteristic value for the sample based on the request and the model to which the request is assigned.

3 . The system of claim 2 , wherein the first randomizing parameter is user ID and the first randomizing function is to assign the request by even or odd number in a last digit of the user ID.

4 . The system of claim 1 , wherein to determine the average difference based on the average first sample subset characteristic values and the average second sample subset characteristic values, the one or more processors are directed to:
determine a first evaluation parameter related to central tendency of the average first sample subset characteristic values; determine a second evaluation parameter related to the central tendency of the average second sample subset characteristic values; determine the average difference based on the first evaluation parameter and the second evaluation parameter.

5 . The system of claim 4 , wherein to determine the significance level based on the average first sample subset characteristic values and the average second sample subset characteristic values, the one or more processors are directed to:
determine a third evaluation parameter related to the central tendencies of the average first sample subset characteristic values and the average second sample subset characteristic values; determine a first error based on difference between the first evaluation parameter and the third evaluation parameter and difference between the second evaluation parameter and the third evaluation parameter; determine a second error based on difference between the average first sample subset characteristic value and the third evaluation parameter and difference between the average second sample subset characteristic value and the third evaluation parameter; and, determine the significance level based on the first error and the second error.

6 . The system of claim 5 , wherein to determine the second error, the one or more processors are further directed to:
determine a degree of freedom based on total number of the first sample subsets and the second sample subsets; and determine the second error based on the degree of freedom.

7 . The system of claim 6 , wherein to determine the confidence interval, the one or more processors are directed to:
obtain a degree of confidence; determine the confidence interval associated with the degree of confidence based on the average difference, the degree of freedom and the second error.

8 . The system of claim 7 , wherein to determine the confidence interval, the one or more processors are directed to:
determine the confidence interval associated with the degree of confidence based on Student's t-distribution.

9 . A method for model evaluation, comprising:
(a) obtaining, by at least one computer, a first sample set and a second sample set, wherein:
(i) the first sample set includes a plurality of first samples based on a first model,
(ii) the second sample set includes a plurality of second samples based on a second model, and
(iii) each of the first and second samples includes a characteristic value,
(b) dividing, by the at least one computer, the first sample set into a plurality of first sample subsets, each first sample subset providing an average first sample subset characteristic value; (c) dividing, by the at least one computer, the second sample set into a plurality of second sample subsets; each second sample subset providing an average second sample subset characteristic value; (d) determining, by the at least one computer, a final model between the first model and the second model based on an average difference, a significance level, and a confidence interval between the first model and the second model, wherein the average difference, the significance level, and the confidence interval are based on the average first sample subset characteristic values and the average second sample subset characteristic values.

10 . The method of claim 9 , wherein obtaining the first sample set and the second sample set, for each sample, includes:
obtaining a request associated with a first randomizing parameter; assigning the request to the first model or the second model based on the first randomizing parameter by using a first randomizing function; and generating the first characteristic value for the sample based on the request and the model to which the request is assigned.

11 . The method of claim 10 , wherein the first randomizing parameter is user ID and the first randomizing function is to assign the request by even or odd number in a last digit of the user ID.

12 . The method of claim 9 , wherein determining the average difference based on the average first sample subset characteristic values and the average second sample subset characteristic values includes:
determining a first evaluation parameter related to central tendency of the average first sample subset characteristic values; determining a second evaluation parameter related to the central tendency of the average second sample subset characteristic values; determining the average difference based on the first evaluation parameter and the second evaluation parameter.

13 . The method of claim 12 , wherein determining the significance level based on the average first sample subset characteristic values and the average second sample subset characteristic values includes:
determining a third evaluation parameter related to the central tendencies of the average first sample subset characteristic values and the average second sample subset characteristic values; determining a first error based on difference between the first evaluation parameter and the third evaluation parameter and difference between the second evaluation parameter and the third evaluation parameter; determining a second error based on difference between the average first sample subset characteristic value and the third evalu

Assignees

• Beijing Didi Infinity Technology & Dev Co Ltd

Inventors

• Zhang Lingyu

Classifications

• G06Q10/0637 Primary

  Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals · CPC title

• G06F11/3447 Primary

  Performance evaluation by modeling · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title