Embedding multi-modal time series and text data

Patent metadata

Field	Value
Publication number	US-2022012274-A1
Application number	US-202117370498-A
Country	US
Kind code	A1
Filing date	Jul 8, 2021
Priority date	Jul 13, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

Methods and systems of training and using a neural network model include training a time series embedding model and a text embedding model with unsupervised clustering to translate time series and text, respectively, to a shared latent space. The time series embedding model and the text embedding model are further trained using semi-supervised clustering that samples training data pairs of time series information and associated text for annotation.

First claim

What is claimed is:

1 . A method of training a neural network, comprising:
training, using a hardware processor, a time series embedding model and a text embedding model with unsupervised clustering to translate time series and text, respectively, to a shared latent space; and training the time series embedding model and the text embedding model further using semi-supervised clustering that samples training data pairs of time series information and associated text for annotation.

2 . The method of claim 1 , wherein the training data pairs each include a time series segment and an associated text.

3 . The method of claim 2 , wherein the associated text describes circumstances relating to the time series segment.

4 . The method of claim 1 , wherein training using semi-supervised clustering includes annotating the sampled pairs to indicate constraints.

5 . The method of claim 4 , wherein the constraints are selected from the group consisting of a “must-link” constraint and a “cannot-link” constraint.

6 . A method of querying a time series database, comprising:
transforming a query to an embedded vector in a multi-modal shared latent space that encodes time series information and textual information; identifying a feature vector in the multi-modal shared latent space, stored in a time series dataspace, that matches the embedded vector, and that is associated with a data type complementary to the query; and returning data associated with the identified feature vector, responsive to the query.

7 . The method of claim 6 , wherein the query includes a text information, and the feature vector is associated with time series information.

8 . The method of claim 7 , wherein the associated text describes circumstances relating to the time series segment.

9 . The method of claim 7 , wherein the query further includes a time series segment.

10 . The method of claim 6 , wherein the query includes time series information, and the feature vector is associated with text information.

11 . The method of claim 6 , wherein identifying the feature vector that matches the embedded vector includes identifying a nearest neighbor according to Euclidean distance within in the multi-modal shared latent space.

12 . A system for training a neural network, comprising:
a hardware processor; and a memory that stores a computer program product, which, when executed by the hardware processor, causes the hardware processor to:
train a time series embedding model and a text embedding model using unsupervised clustering to translate time series and text, respectively, to a multi-modal shared latent space;
train the time series embedding model and the text embedding model further using semi-supervised clustering that samples training data pairs of time series information and associated text for annotation;
transform a query to an embedded vector in the multi-modal shared latent space that encodes time series information and textual information;
identifying a feature vector in the multi-modal shared latent space, stored in a time series dataspace, that matches the embedded vector, and that is associated with a data type complementary to the query; and
returning data associated with the identified feature vector, responsive to the query.

13 . The system of claim 12 , wherein the training data pairs each include a time series segment and an associated text.

14 . The system of claim 13 , wherein the associated text of each pair describes circumstances relating to the respective time series segment.

15 . The system of claim 13 , wherein the computer program product further causes the hardware processor to annotate the sampled pairs to indicate constraints during semi-supervised clustering.

16 . The system of claim 15 , wherein the constraints are selected from the group consisting of a “must-link” constraint and a “cannot-link” constraint.

17 . The system of claim 12 , wherein the query includes a text information, and the feature vector is associated with time series information.

18 . The system of claim 16 , wherein the query further includes a time series segment.

19 . The system of claim 12 , wherein the query includes time series information, and the feature vector is associated with text information.

20 . The system of claim 12 , wherein the computer program product further causes the hardware processor to identify a nearest neighbor of the embedded vector according to Euclidean distance within in the multi-modal shared latent space.

Assignees

• Nec Lab America Inc

Inventors

• Chen Yuncong
• Song Dongjin
• Lumezanu Cristian
• Chen Haifeng
• Mizoguchi Takehiko
• Zhang Xuchao

Classifications

• G06V10/62

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06N3/045

  Combinations of networks · CPC title