Systems and methods for sequential event prediction with noise-contrastive estimation for marked temporal point process

Patent metadata

Field	Value
Publication number	US-12014267-B2
Application number	US-201916511505-A
Country	US
Kind code	B2
Filing date	Jul 15, 2019
Priority date	Jul 13, 2018
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

Embodiments for systems and methods of sequential event prediction with noise-contrastive estimation for marked temporal point process are disclosed.

First claim

What is claimed is:

1. A method of improving the training of marked temporal point process models, comprising
utilizing a processor in communication with a tangible storage medium storing instructions that are executed by the processor to perform operations comprising:
accessing a model that expresses sequential events using a marked temporal point process, the model configured to formulate a prediction of a future event and associated future event characteristics based on sequential event data applied to the model;
training the model, by:
identifying event samples P d from a data distribution and initial noise samples P n from a noise distribution associated with the model,
generating additional noise samples from the noise distribution, by
computing values for a predicted time and a predicted mark by using as inputs at least a history sequence i and a parametric model p θ ;
generating, iteratively, a noise sample using the noise distribution and inputting at least a predicted time t and predicted mark x, and
adding an additional noise to the noise sample, wherein the additional noise is sampled from a second noise distribution with a variance, wherein a value of the variance associated with the second noise distribution increases with respect to a number of iterations, and

utilizing the event samples, the additional noise samples, and a noise contrastive loss function to re-parameterize the model; and
updating the noise distribution to become increasingly similar to the data distribution the model accesses more information from the data distribution to make classification between the noise samples and the event samples increasingly difficult for the model to learn.

2. The method of claim 1 , wherein the probability density function outputs an adaptive Gaussian noise distribution.

3. The method of claim 1 , wherein a neural deep learning model maps an observed history of the history sequence i to a vector representation i .

4. The method of claim 1 , wherein the event samples p d are continuous joint distributions of a time value t and predicted mark x.

5. The method of claim 1 , further comprising generating at least one noise event for an observed event.

6. The method of claim 1 , wherein a plurality of dense layers are applied to project a raw input into a multi-dimensional space.

7. A processor, configured to:
access a model that expresses sequential events using a marked temporal point process, the model configured to formulate a prediction of a future event and associated future event characteristics based on sequential event data applied to the model;
train the model using adaptive noise sample generation, by:
identifying event samples p d from a data distribution and initial noise samples p n from a noise distribution associated with the model,
generating additional noise samples from the noise distribution, by
computing values for a predicted time and a predicted mark by using as inputs at least a history sequence i and a parametric model p θ ; and
generating, iteratively, a noise sample using the noise distribution and inputting at least a predicted time t and predicted mark x,
adding an additional noise to the noise sample, wherein the additional noise is sampled from a second noise distribution with a variance, wherein a value of the variance associated with the second noise distribution increases with respect to a number of iterations, and

utilizing the event samples, the additional noise samples, and a noise contrastive loss function to re-parameterize the model to adaptively push noise distribution associated with p n towards p d as the model accesses more information from the data distribution.

8. The processor of claim 7 , wherein the probability density function outputs an adaptive Gaussian noise distribution.

9. The processor of claim 7 , further configured to employ a neural deep learning model that maps an observed history of the history sequence i to a vector representation h i .

10. The processor of claim 7 , wherein the event samples p d are continuous joint distributions of a time value t and predicted mark x.

11. The processor of claim 7 , further configured to generate at least one noise event for an observed event.

12. The processor of claim 7 , further configured to apply a plurality of dense layers to project a raw input into a multi-dimensional space.

13. A processor, configured to:
access sequential event data; and
formulate, by execution of a model in view of the sequential event data, a prediction of a future event and associated future event characteristics, the model configured to express sequential events using a marked temporal point process, the model configured for noise-contrastive estimation wherein true and noise samples refer to events observed in a data distribution pa and a specified noise distribution p n ,
wherein the model is trained using adaptive noise sample generation to make the noise distribution as close as possible to the data distribution so that the model learns to discriminate better by solving more difficult classification problems.

14. The processor of claim 13 , wherein the model is trained using the adaptive noise sample generation by identifying event samples p d from the data distribution and initial noise samples p n from the noise distribution associated with the model, generating additional noise samples from the noise distribution, and utilizing the event samples, the additional noise samples, and a noise contrastive loss function to re-parameterize the model to adaptively push noise distribution associated with p n towards p d as the model accesses more information from the data distribution.

15. The processor of claim 13 , wherein the model is a recurrent neural network (RNN).

Assignees

• Guo Ruocheng
• Li Jundong
• Liu Huan
• Univ Arizona State

Inventors

• Guo Ruocheng
• Li Jundong
• Liu Huan

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/048

  Activation functions · CPC title