Embedding multi-modal time series and text data

Patent metadata

Field	Value
Publication number	US-11741146-B2
Application number	US-202117370498-A
Country	US
Kind code	B2
Filing date	Jul 8, 2021
Priority date	Jul 13, 2020
Publication date	Aug 29, 2023
Grant date	Aug 29, 2023

Abstract

Methods and systems of training and using a neural network model include training a time series embedding model and a text embedding model with unsupervised clustering to translate time series and text, respectively, to a shared latent space. The time series embedding model and the text embedding model are further trained using semi-supervised clustering that samples training data pairs of time series information and associated text for annotation.

First claim

What is claimed is:

1. A method of querying a time series database, comprising:
transforming a query to an embedded vector in a multi-modal shared latent space that encodes time series info, illation and textual information;
identifying a feature vector in the multi-modal shared latent space, stored in a time series dataspace, that matches the embedded vector, and that is associated with a data type complementary to the query; and
returning data associated with the identified feature vector, responsive to the query.

2. The method of claim 1 , wherein the query includes a text information, and the feature vector is associated with time series information.

3. The method of claim 2 , wherein the associated text describes circumstances relating to the time series segment.

4. The method of claim 2 , wherein the query further includes a time series segment.

5. The method of claim 1 , wherein the query includes time series information, and the feature vector is associated with text information.

6. The method of claim 1 , wherein identifying the feature vector that matches the embedded vector includes identifying a nearest neighbor according to Euclidean distance within in the multi-modal shared latent space.

7. A system for training a neural network, comprising:
a hardware processor; and
a memory that stores a computer program product, which, when executed by the hardware processor, causes the hardware processor to:
train a time series embedding model and a text embedding model using unsupervised clustering to translate time series and text, respectively, to a multi-modal shared latent space;
train the time series embedding model and the text embedding model further using semi-supervised clustering that samples training data pairs of time series information and associated text for annotation;
transform a query to an embedded vector in the multi-modal shared latent space that encodes time series information and textual information;
identifying a feature vector in the multi-modal shared latent space, stored in a time series dataspace, that matches the embedded vector, and that is associated with a data type complementary to the query; and
returning data associated with the identified feature vector, responsive to the query.

8. The system of claim 7 , wherein the training data pairs each include a time series segment and an associated text.

9. The system of claim 8 , wherein the associated text of each pair describes circumstances relating to the respective time series segment.

10. The system of claim 8 , wherein the computer program product further causes the hardware processor to annotate the sampled pairs to indicate constraints during semi-supervised clustering.

11. The system of claim 10 , wherein the constraints are selected from the group consisting of a “must-link” constraint and a “cannot-link” constraint.

12. The system of claim 7 , wherein the query includes a text information, and the feature vector is associated with time series information.

13. The system of claim 11 , wherein the query further includes a time series segment.

14. The system of claim 7 , wherein the query includes time series information, and the feature vector is associated with text information.

15. The system of claim 7 , wherein the computer program product further causes the hardware processor to identify a nearest neighbor of the embedded vector according to Euclidean distance within in the multi-modal shared latent space.

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Chen Yuncong
• Song Dongjin
• Lumezanu Cristian
• Chen Haifeng
• Mizoguchi Takehiko
• Zhang Xuchao

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/091

  Active learning · CPC title

• G06F16/355 Primary

  Creation or modification of classes or clusters · CPC title