Device and a method for processing data sequences using a convolutional neural network

Patent metadata

Field	Value
Publication number	US-11521059-B2
Application number	US-201916373939-A
Country	US
Kind code	B2
Filing date	Apr 3, 2019
Priority date	Apr 23, 2018
Publication date	Dec 6, 2022
Grant date	Dec 6, 2022

Abstract

A device for processing data sequences by means of a convolutional neural network is configured to carry out the following steps: receiving an input sequence comprising a plurality of data items captured over time using a sensor, each of said data items comprising a multi-dimensional representation of a scene, generating an output sequence representing the input sequence processed item-wise by the convolutional neural network, wherein generating the output sequence comprises: generating a grid-generation sequence based on a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the output sequence or the grid-generation sequence, generating a sampling grid on the basis of the grid-generation sequence, generating an intermediate output sequence by sampling from the past portion of the output sequence according to the sampling grid, and generating the output sequence based on a weighted combination of the intermediate output sequence and the input sequence.

First claim

We claim:

1. A device for processing data sequences comprising a convolutional neural network, the device being configured to receive an input sequence comprising a plurality of data items captured over time, each data item of the plurality of data items comprising a multi-dimensional representation of a scene, the convolutional neural network being configured to:
generate an output sequence representing the input sequence processed item-wise by the convolutional neural network, the convolutional neural network comprising a sampling unit configured to generate an intermediate output sequence by sampling from a past portion of the output sequence according to a sampling grid;
generate the sampling grid item-wise on a basis of a grid-generation sequence, wherein the grid-generation sequence is based on a combination of the input sequence and an intermediate grid-generation sequence representing the past portion of the output sequence or the grid-generation sequence; and
generate the output sequence based on a weighted combination of the intermediate output sequence and the input sequence.

2. The device according to claim 1 , wherein the grid-generation sequence is based on an item-wise combination of the input sequence and the intermediate grid-generation sequence.

3. The device according to claim 1 , wherein:
the intermediate grid-generation sequence is formed by the past portion of the output sequence;
the intermediate grid-generation sequence is formed by the past portion of the output sequence processed with an inner convolutional neural network; or
the intermediate grid-generation sequence is formed by the past portion of the grid-generation sequence processed with an inner convolutional neural network.

4. The device according to claim 1 , wherein the convolutional neural network is configured to generate the sampling grid by processing the grid-generation sequence with at least one inner convolutional neural network.

5. The device according to claim 1 , wherein the convolutional neural network is configured to generate the output sequence by generating a first weighting sequence and a second weighting sequence based on one of:
the input sequence;
the intermediate output sequence,
the intermediate grid-generation sequence;
the grid-generation sequence processed by an inner convolutional network;
generating an intermediate input sequence by processing the input sequence with an inner convolutional neural network;
weighting the intermediate output sequence with the first weighting sequence;
weighting the intermediate input sequence with the second weighting sequence; or
superimposing the weighted intermediate output sequence and the weighted intermediate input sequence.

6. The device according to claim 5 , wherein generating the first weighting sequence and the second weighting sequence comprises:
forming a combination of at least two of the input sequence, the intermediate output sequence, the intermediate grid-generation sequence, or the grid-generation sequence processed by the inner convolutional network; and
forming a processed combination by processing the combination with an inner convolutional neural network.

7. The device according to claim 6 , wherein one of the first weighting sequence or the second weighting sequence is formed by the processed combination and wherein the other of the first weighting sequence or the second weighting sequence is formed by the processed combination subtracted from a constant.

8. The device according to claim 5 , wherein the convolutional neural network is configured to generate the first weighting sequence and the second weighting sequence correspondingly.

9. The device according to claim 1 , wherein the sampling grid comprises a plurality of sampling locations, each sampling location of the plurality of sampling locations being defined by a respective pair of an offset and one of a plurality of data points of a data item of the intermediate output sequence.

10. The device according to claim 1 , wherein each data item of the input sequence comprises a plurality of data points, each data point representing a location in the scene and comprising a plurality of parameters of the location.

11. The device according to claim 1 , wherein each data item of the input sequence is formed by an image comprising a plurality of pixels.

12. The device according to claim 10 , wherein the plurality of parameters of the location comprise coordinates of the location.

13. A system for processing data sequences, the system comprising:
a sensor for capturing a data sequence; and
a device comprising a convolutional neural network, the device being configured to receive an input sequence comprising a plurality of data items captured over time, each data item of the plurality of data items comprising a multi-dimensional representation of a scene; the convolutional neural network being configured to:
generate an output sequence representing the input sequence processed item-wise by the convolutional neural network, the convolutional neural network comprising a sampling unit configured to generate an intermediate output sequence by sampling from a past portion of the output sequence according to a sampling grid;
generate the sampling grid item-wise on a basis of a grid-generation sequence, wherein the grid-generation sequence is based on a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the output sequence or the grid-generation sequence; and
generate the output sequence based on a weighted combination of the intermediate output sequence and the input sequence.

14. The system according to claim 13 , wherein the sensor comprises at least one of a radar sensor, a light-detection-and-ranging sensor, an ultrasonic sensor, or a camera.

15. The system according to claim 13 , wherein the grid-generation sequence is based on an item-wise combination of the input sequence and the intermediate grid-generation sequence.

16. The system according to claim 13 , wherein:
the intermediate grid-generation sequence is formed by the past portion of the output sequence;
the intermediate grid-generation sequence is formed by the past portion of the output sequence processed with an inner convolutional neural network; or
the intermediate grid-generation sequence is formed by the past portion of the grid-generation sequence processed with an inner convolutional neural network.

17. The system according to claim 1 , wherein the convolutional neural network is configured to generate the sampling grid by processing the grid-generation sequence with at least one inner convolutional neural network.

18. The system according to claim 13 , wherein the convolutional neural network is configured to generate the output sequence by generating a first weighting sequence and a second weighting sequence based on one of:
the input sequence;
the intermediate output sequence;
the intermediate grid-generation sequence;
the grid-generation sequence processed by an inner convolutional network;
generating an intermediate input sequence by processing the input sequence with an inner convolutional neural network;
weighting the intermediate output sequence with the first weighting sequence;
weighting the intermediate input sequence with the second weighting sequence; or
superimposing the weighted intermediate output sequence and the weighted intermediate input sequence.

19. The system according to claim 18 , wherein generating the first weighting sequence and the s

Assignees

• Aptiv Tech Ltd

Inventors

• Zhu Weimeng
• Su Yu
• Nunn Christian

Classifications

• G06V10/62

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/70

  using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) · CPC title

• G06N3/044 Primary

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title