Warm start generalized additive mixed-effect (game) framework

Patent metadata

Field	Value
Publication number	US-11106982-B2
Application number	US-201816109411-A
Country	US
Kind code	B2
Filing date	Aug 22, 2018
Priority date	Aug 22, 2018
Publication date	Aug 31, 2021
Grant date	Aug 31, 2021

Abstract

In an example embodiment, a warm-start training solution is used to dramatically reduce the computational resources needed to train when retraining a generalized additive mixed-effect (GAME) model. The problem of retraining time is particularly applicable to GAME models, since these models take much longer to train as the data grows. In the past, the strategy to reduce computational resources during retraining was to use less training data, but this affects the model quality, especially for GAME models, which rely on fine-grained sub-models at, for example, member or item levels. The present solution addresses the computational resources issues without sacrificing GAME model accuracy.

First claim

What is claimed is:

1. A system comprising:
a computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the system to:
divide training data obtained from data in a social networking service into a plurality of intervals based on time as indicated in time stamps in the data, each interval associated with a different value of i successively increasing for each interval;
derive a first set of features from the training data for a first of the plurality of intervals;
feed the first set of features into a generalized additive mixed effect (GAME) model, the GAME model producing a trained fixed effect portion and one or more trained random effects portions;
for each subsequent time interval:
retrain the fixed effect portion using fixed effect coefficients from a version of the fixed effect portion trained using data from the interval associated with time i−1 while converging on training data contained in the interval associated with time i;
for each of one or more random effects contained in the training data in the interval associated with time i:
reuse a version of a random effect portion from the GAME model trained using data from the interval associated with time i−1 if a number of data points corresponding to the random effect in the training data in the interval associated with time i does not transgress a first threshold.

2. The system of claim 1 , wherein the instructions further cause the system to:
for each subsequent time interval:
for each of one or more random effects contained in the training data in the interval associated with time i:
train or retrain the random effect portion using training data from the interval associated with time i if the number of data points corresponding to the random effect in the training data in the interval associated with time i transgresses a first threshold.

3. The system of claim 2 , wherein the training or retraining includes minimizing ΣL(x, β)+λ*∥β−b∥ 2 where x is a feature vector associated with one or more features extracted from the training data in the interval associated with time i, β is a learned coefficient vector from the a current version of the GAME model, L is a loss function, λ is a regularization weight, where b is the coefficient of the existing model.

4. The system of claim 1 , wherein the retraining is repeated until convergence, wherein convergence occurs when a subsequent version of the GAME model differs from an immediately preceding version of the GAME model by less than a second threshold.

5. The system of claim 1 , wherein the retraining further comprises updating fixed effect offsets of the fixed effect portion.

6. The system of claim 2 , wherein the training or retraining further comprises updating random effect offsets of the random effect portion.

7. The system of claim 1 , wherein the first threshold is dynamically set based on attributes of a first user to which a prediction to be output by the GAME model is associated.

8. A computerized method comprising:
dividing training data obtained from data in a social networking service into a plurality of intervals based on time as indicated in time stamps in the data, each interval associated with a different value of i successively increasing for each interval;
deriving a first set of features from the training data for a first of the plurality of intervals;
feeding the first set of features into a generalized additive mixed effect (GAME) model, the GAME model producing a trained fixed effect portion and one or more trained random effects portions;
for each subsequent time interval:
retraining the fixed effect portion using fixed effect coefficients from a version of the fixed effect portion trained using data from the interval associated with time i−1 while converging on training data contained in the interval associated with time i;
for each of one or more random effects contained in the training data in the interval associated with time i:
reusing a version of a random effect portion from the GAME model trained using data from the interval associated with time i−1 if a number of data points corresponding to the random effect in the training data in the interval associated with time i does not transgress a first threshold.

9. The method of claim 8 , further comprising:
for each of one or more random effects contained in the training data in the interval associated with time i:
training or retraining the random effect portion using training data from the interval associated with time i if the number of data points corresponding to the random effect in the training data in the interval associated with time i transgresses a first threshold.

10. The method of claim 9 , wherein the training or retraining includes minimizing ΣL(x, β)+λ*∥β−b∥ 2 where x is a feature vector associated with one or more features extracted from the training data in the interval associated with time i, β is a learned coefficient vector from the a current version of the GAME model, L is a loss function, λ is a regularization weight, where b is the coefficient of the existing model.

11. The method of claim 8 , wherein the retraining is repeated until convergence, wherein convergence occurs when a subsequent version of the GAME model differs from an immediately preceding version of the GAME model by less than a second threshold.

12. The method of claim 8 , wherein the retraining further comprises updating fixed effect offsets of the fixed effect portion.

13. The method of claim 9 , wherein the training or retraining further comprises updating random effect offsets of the random effect portion.

14. The method of claim 8 , wherein the first threshold is dynamically set based on attributes of a first user to which a prediction to be output by the GAME model is associated with.

15. A non-transitory machine-readable storage medium comprising instructions which, when implemented by one or more machines, cause the one or more machines to perform operations comprising:
dividing training data obtained from data in a social networking service into a plurality of intervals based on time as indicated in time stamps in the data, each interval associated with a different value of i successively increasing for each interval;
deriving a first set of features from the training data for a first of the plurality of intervals;
feeding the first set of features into a generalized additive mixed effect (GAME) model, the GAME model producing a trained fixed effect portion and one or more trained random effects portions;
for each subsequent time interval:
retraining the fixed effect portion using fixed effect coefficients from a version of the fixed effect portion trained using data from the interval associated with time i−1 while converging on training data contained in the interval associated with time i;
for each of one or more random effects contained in the training data in the interval associated with time i:
reusing a version of a random effect portion from the GAME model trained using data from the interval associated with time i−1 if a number of data points corresponding to the random effect in the training data in the interval associated with time i does not transgress a first threshold.

16. The non-transitory machine-readable storage medium of claim 15 , further comprising:
for each of one or more random effects contained in the training data in the interval associated with time i:
training or retraining the random effect portion using training data from the interval associated with time i if the number of data points corresponding to t

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ma Yiming
• Shelkovnykov Alex
• Fleming Josh
• Chen Bee-Chung
• Long Bo

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F16/9535 Primary

  Search customisation based on user profiles and personalisation · CPC title

• G06N7/00

  Computing arrangements based on specific mathematical models · CPC title

• G06Q10/1053

  Employment or hiring · CPC title

• G06N3/126

  Evolutionary algorithms, e.g. genetic algorithms or genetic programming · CPC title