Time series data adaptation and sensor fusion systems, methods, and apparatus

Patent metadata

Field	Value
Publication number	US-10410113-B2
Application number	US-201614995688-A
Country	US
Kind code	B2
Filing date	Jan 14, 2016
Priority date	Jan 14, 2016
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

Systems, methods, and apparatus for time series data adaptation, including sensor fusion, are disclosed. For example, a system includes a variational inference machine, a sequential data forecast machine including a hidden state, and a machine learning model. The sequential data forecast machine exports a version of the hidden state. The variational inference machine receives as inputs time series data and the version of the hidden state, and outputs a time dependency infused latent distribution. The sequential data forecast machine obtains the version of the hidden state, receives as inputs the time series data and the time dependency infused latent distribution, and updates the hidden state based on the time series data, the time dependency infused latent distribution, and the version of the hidden state to generate a second version of the hidden state. The time dependency infused latent distribution is input into the machine learning model, which outputs a result.

First claim

The invention is claimed as follows:

1. A computer-implemented system comprising:
a variational inference machine including a neural network that includes a first input layer, a first plurality of hidden layers, and a first output layer;
a sequential data forecast machine including a neural network that includes a second input layer and a second plurality of hidden layers, the second plurality of hidden layers including a hidden state; and
a machine learning model including a neural network, wherein the variational inference machine, the sequential data forecast machine, and the machine learning model are configured to iteratively perform:
exporting, from the second plurality of hidden layers, a first version of the hidden state, which includes time dependency information,
receiving, at the first input layer, (i) time series data of a first time interval from at least a first sensor and (ii) the first version of the hidden state,
outputting, by the first output layer, a time dependency infused latent distribution that is generated based on the time series data of the first time interval and the first version of the hidden state,
obtaining, by the second plurality of hidden layers, the first version of the hidden state,
receiving, at the second input layer, (i) the time series data of the first time interval and (ii) the time dependency infused latent distribution from the variational inference machine,
updating the hidden state in the second plurality of hidden layers based on the time series data of the first time interval, the time dependency infused latent distribution, and the first version of the hidden state to generate a second version of the hidden state,
inputting the time dependency infused latent distribution into the machine learning model, and
outputting, from the machine learning model, a result of the first time interval based on the time dependency infused latent distribution.

2. The computer-implemented system of claim 1 , wherein the result causes an instruction to be sent to an actuator to perform an actuation.

3. The computer-implemented system of claim 2 , wherein the actuation is at least one of steering or braking.

4. The computer-implemented system of claim 1 , wherein the time series data, which is output from a first sensor, is unavailable at the first time interval.

5. The computer-implemented system of claim 4 , wherein the result is fault tolerant to intermittent unavailability of the first sensor at the first time interval.

6. The computer-implemented system of claim 1 , wherein the time series data is multi-dimensional data from one of an image sensor, a lidar sensor, or a radar sensor.

7. The computer-implemented system of claim 1 , wherein the time series data includes a first time series of a first modality and a second time series of a second modality, and the time dependency infused latent distribution is a multi-modal time dependency infused latent distribution.

8. The computer-implemented system of claim 1 , further comprising a first sensor and a second sensor, wherein the first sensor senses a first type of data of a first modality, and the second sensor senses a second type of data of a second modality.

9. The computer-implemented system of claim 8 , wherein the first modality is radar and the second modality is ultrasound.

10. The computer-implemented system of claim 8 , further comprising a third sensor, which senses a third type of data of a third modality.

11. The computer-implemented system of claim 10 , wherein the first modality is radar, the second modality is ultrasound, and the third modality is lidar.

12. The computer-implemented system of claim 1 , wherein the first input layer and the second input layer receive a first version of a second hidden state from a second sequential data forecast machine.

13. The computer-implemented system of claim 12 , wherein time dependency infused latent distribution is generated based on the first version of the second hidden state from the second sequential data forecast machine.

14. The computer-implemented system of claim 13 , wherein the second sequential data forecast machine is a multi-modal sequential data forecast machine.

15. The computer-implemented claim 1 , wherein the first input layer includes a first sublayer and a second sublayer, wherein the time series data is input into the first sublayer, and the hidden state is input into the second sublayer.

16. The computer-implemented system of claim 1 , wherein the sequential data forecast machine includes a recurrent neural network.

17. The computer-implemented system of claim 16 , wherein the recurrent neural network includes a second output layer that outputs a prediction of future time series data.

18. The computer-implemented system of claim 1 , further comprising a variational generation machine that generates an output representative of the time series data received by the first input layer.

19. The computer-implemented system of claim 1 , wherein the computer-implemented system is implemented in an edge device.

20. The computer-implemented system of claim 1 , wherein the computer-implemented system is implemented in a vehicle.

21. The computer-implemented system of claim 1 , wherein a new version of the hidden state is generated in a different time interval at least once every 30 milliseconds.

22. The computer-implemented system of claim 21 , wherein the machine learning model executes at least once every 30 milliseconds using a new version of the time dependency infused latent distribution.

23. The computer-implemented system of claim 1 , wherein the time series data includes image data.

24. The computer-implemented system of claim 1 , wherein the time series data includes sound data.

25. The computer-implemented system of claim 1 , wherein the time series data includes acceleration data.

26. The computer-implemented system of claim 1 . wherein the time series data is binary data.

27. The computer-implemented system of claim 1 , wherein the result is at least one of a forecast, a prediction, a classification, a clustering, an anomaly detection, or a recognition.

28. A computer-implemented method comprising:
exporting, from a plurality of hidden layers of a sequential data forecast machine including a neural network, a first version of a hidden state that includes time dependency information;
receiving, at a first input layer of a variational inference machine including a neural network, (i) time series data of a first time interval from at least a first sensor and (ii) the first version of the hidden state;
outputting, by a first output layer of the variational inference machine, a time dependency infused latent distribution that is generated based on the time series data of the first time interval and the first version of the hidden state;
obtaining, by the plurality of hidden layers, the first version of the hidden state;
receiving, at a second input layer of the sequential data forecast machine, (i) the time series data of the first time interval and (ii) the time dependency infused latent distribution from the variational inference machine;
updating the hidden state in the plurality of hidden layers based on the time series data of the first time interval, the time dependency infused latent distribution, and the first version of the hidden state to generate a second version of the hidden s

Assignees

• Preferred Networks Inc

Inventors

• Clayton Justin B
• Okanohara Daisuke
• Hido Shohei

Classifications

• G06N3/044 Primary

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0454 Primary

  Physics · mapped topic

• G06N3/0445

  Physics · mapped topic

• G06N3/0455 Primary

  Auto-encoder networks; Encoder-decoder networks · CPC title