Systems and methods for time series analysis using attention models

Patent metadata

Field	Value
Publication number	US-11699079-B2
Application number	US-202016748985-A
Country	US
Kind code	B2
Filing date	Jan 22, 2020
Priority date	Jan 22, 2019
Publication date	Jul 11, 2023
Grant date	Jul 11, 2023

Abstract

A system for time series analysis using attention models is disclosed. The system may capture dependencies across different variables through input embedding and may map the order of a sample appearance to a randomized lookup table via positional encoding. The system may capture capturing dependencies within a single sequence through a self-attention mechanism and determine a range of dependency to consider for each position being analyzed. The system may obtain an attention weighting to other positions in the sequence through computation of an inner product and utilize the attention weighting to acquire a vector representation for a position and mask the sequence to enable causality. The system may employ a dense interpolation technique for encoding partial temporal ordering to obtain a single vector representation and a linear layer to obtain logits from the single vector representation. The system may use a type dependent final prediction layer.

First claim

What is claimed is:

1. A computer-based method for analyzing and modeling a multivariate time series data based on an attention computation, the method comprising:
capturing dependencies across different variables through input embedding;
mapping an order of a sample appearance to a randomized lookup table via positional encoding;
capturing dependencies within a plurality of self-attention mechanisms, each self-attention mechanism of the plurality of self-attention mechanisms capturing dependencies within a single sequence of each self-attention mechanism;
determining a range of dependency to consider for each position being analyzed within the single sequence of each self-attention mechanism;
obtaining a plurality of attention weightings to other positions within the single sequence through computation of an inner product, each of the plurality of attention weightings obtained within the single sequence of each self-attention mechanism;
utilizing the plurality of attention weightings to acquire a plurality of vector representations for a position;
masking the single sequence of each self-attention mechanism to enable causality;
employing a dense interpolation technique for encoding partial temporal ordering to obtain a single vector representation from the plurality of vector representations;
applying a linear layer to obtain logits from the single vector representation; and
applying a final prediction layer whose type depends on a specific task.

2. The method of claim 1 , wherein the self-attention mechanism is a masked multi-head mechanism.

3. The method of claim 1 , wherein analysis and modeling of the multivariate time series data is fully parallelizable.

4. The method of claim 1 , wherein the specific task is a binary classification problem and the final prediction layer includes a softmax layer.

5. A system for analyzing and modeling a multivariate time series data based on an attention computation, comprising:
a controller; and
a tangible, non-transitory memory configured to communicate with the controller, the tangible, non-transitory memory having instructions stored thereon that, in response to execution by the controller, cause the controller to perform operations comprising:
capturing, by the controller, dependencies across different variables through input embedding;
mapping, by the controller, an order of a sample appearance to a randomized lookup table via positional encoding;
capturing, by the controller, dependencies within a plurality of a self-attention mechanisms, each self-attention mechanism of the plurality of self-attention mechanisms capturing dependencies within a single sequence of each self-attention mechanism;
determining, by the controller, a range of dependency to consider for each position being analyzed within the single sequence of each self-attention mechanism;
obtaining, by the controller, a plurality of attention weightings to other positions within the single sequence through computation of an inner product, each of the plurality of attention weightings obtained within the single sequence of each self-attention mechanism;
utilizing, by the controller, the plurality of attention weightings to acquire a plurality of vector representations for a position;
masking, by the controller, the single sequence of each self-attention mechanism to enable causality;
employing, by the controller, a dense interpolation technique for encoding partial temporal ordering to obtain a single vector representation from the plurality of vector representations;
utilizing, by the controller, a linear layer to obtain logits from the single vector representation; and
using, by the controller, a final prediction layer whose type depends on a specific task.

6. The system of claim 5 , wherein the self-attention mechanism is a masked multi-head mechanism.

7. The system of claim 5 , wherein analysis and modeling of the multivariate time series data is fully parallelizable.

8. The system of claim 5 , wherein the specific task is a binary classification problem and the final prediction layer includes a softmax layer.

9. An article of manufacture including a tangible, non-transitory computer-readable storage medium having instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations comprising:
capturing, by the processor, dependencies across different variables of a multivariate time series data through input embedding;
mapping, by the processor, an order of a sample appearance to a randomized lookup table via positional encoding;
capturing, by the processor, dependencies within a plurality of self-attention mechanisms, each self-attention mechanism of the plurality of self-attention mechanisms capturing dependencies within a single sequence of each self-attention mechanism;
determining, by the processor, a range of dependency to consider for each position being analyzed within the single sequence of each self-attention mechanism;
obtaining, by the processor, a plurality of attention weightings to other positions within the single sequence through computation of an inner product, each of the plurality of attention weightings obtained within the single sequence of each self-attention mechanism;
utilizing, by the processor, the plurality of attention weightings to acquire a plurality of vector representations for a position;
masking, by the processor, the single sequence of each self-attention mechanism to enable causality;
employing, by the processor, a dense interpolation technique for encoding partial temporal ordering to obtain a single vector representation from the plurality of vector representations;
utilizing, by the processor, a linear layer to obtain logits from the single vector representation; and
using, by the processor, a final prediction layer whose type depends on a specific task.

10. The article of manufacture of claim 9 , wherein the self-attention mechanism is a masked multi-head mechanism.

11. The article of manufacture of claim 9 , wherein the operations by the processor are fully parallelizable.

12. The article of manufacture of claim 9 , wherein the specific task is a binary classification problem and the final prediction layer includes a softmax layer.

Assignees

• Spanias Andreas
• Song Huan
• Thiagarajan Jayaraman J
• Rajan Deepta
• Univ Arizona State
• L Livermore Nat Security Llc

Inventors

• Spanias Andreas
• Song Huan
• Thiagarajan Jayaraman J
• Rajan Deepta

Classifications

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06F2218/00

  Aspects of pattern recognition specially adapted for signal processing · CPC title

• G06F18/213

  Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title