Device and a method for processing data sequences using a convolutional neural network

Patent metadata

Field	Value
Publication number	US-11804026-B2
Application number	US-202218047680-A
Country	US
Kind code	B2
Filing date	Oct 19, 2022
Priority date	Apr 23, 2018
Publication date	Oct 31, 2023
Grant date	Oct 31, 2023

Abstract

A device for processing data sequences by means of a convolutional neural network is configured to carry out the following steps: receiving an input sequence comprising a plurality of data items captured over time using a sensor, each of said data items comprising a multi-dimensional representation of a scene, generating an output sequence representing the input sequence processed item-wise by the convolutional neural network, wherein generating the output sequence comprises: generating a grid-generation sequence based on a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the output sequence or the grid-generation sequence, generating a sampling grid on the basis of the grid-generation sequence, generating an intermediate output sequence by sampling from the past portion of the output sequence according to the sampling grid, and generating the output sequence based on a weighted combination of the intermediate output sequence and the input sequence.

First claim

What is claimed is:

1. A device comprising:
a convolutional neural network configured to:
receive, from a sensor, an input sequence comprising a plurality of data items captured over time in a scene of a vehicle;
generate an output sequence representing the input sequence processed item-wise by the convolutional neural network by at least:
generating an intermediate output sequence by sampling from a past portion of the output sequence according to a sampling grid;
generating, based on a grid-generation sequence, the sampling grid item-wise, the grid-generation sequence being a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the grid-generation sequence;
generating, based on the intermediate output sequence, the grid-generation sequence, the intermediate grid-generation sequence, or the input sequence, a first weighting sequence and a second weighting sequence;
generating, based on the first weighting sequence and the second weighting sequence, a weighted combination of the intermediate output sequence and the input sequence; and
generating, based on the weighted combination of the intermediate output sequence and the input sequence, the output sequence.

2. The device of claim 1 , wherein the convolutional neural network is further configured to generate the output sequence by:
generating a weighted input sequence by weighting an intermediate input sequence with the first weighting sequence, the intermediate input sequence formed by processing the input sequence with an inner convolutional neural network;
generating a weighted intermediate output sequence by weighting the intermediate output sequence with the second weighting sequence; and
superimposing the weighted input sequence and the weighted intermediate output sequence to generate the weighted combination of the intermediate output sequence and the input sequence.

3. The device of claim 2 , wherein the convolutional neural network is configured to generate the first weighting sequence or the second weighting sequence based on the intermediate output sequence.

4. The device of claim 3 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the intermediate output sequence.

5. The device of claim 3 , wherein:
generating the first weighting sequence is based on forming a combination of the grid-generation sequence and the intermediate output sequence; and
generating the second weighting sequence is based on forming a combination of the input sequence, the grid-generation sequence, and the intermediate output sequence.

6. The device of claim 3 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence, the grid-generation sequence, and the intermediate output sequence.

7. The device of claim 2 , wherein the convolutional neural network is configured to generate the first weighting sequence and the second weighting sequence without using the intermediate output sequence.

8. The device of claim 7 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the grid-generation sequence.

9. The device of claim 7 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the intermediate grid-generation sequence.

10. The device of claim 7 , wherein the convolutional neural network is further configured to generate the grid-generation sequence by:
processing the superimposed weighted input sequence and the weighted intermediate output sequence with the inner convolutional neural network.

11. A system comprising:
at least one sensor for capturing a data sequence; and
a device comprising:
a convolutional neural network configured to:
receive, from the at least one sensor, an input sequence comprising a plurality of data items captured over time in a scene of a vehicle;
generate an output sequence representing the input sequence processed item-wise by the convolutional neural network by at least:
generating an intermediate output sequence by sampling from a past portion of the output sequence according to a sampling grid;
generating, based on a grid-generation sequence, the sampling grid item-wise, the grid-generation sequence being a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the grid-generation sequence;
generating, based on the intermediate output sequence, the grid-generation sequence, the intermediate grid-generation sequence, or the input sequence, a first weighting sequence and a second weighting sequence;
generating, based on the first weighting sequence and the second weighting sequence, a weighted combination of the intermediate output sequence and the input sequence; and
generating, based on the weighted combination of the intermediate output sequence and the input sequence, the output sequence.

12. The system of claim 11 , wherein the convolutional neural network is further configured to generate the output sequence by:
generating a weighted input sequence by weighting an intermediate input sequence with the first weighting sequence, the intermediate input sequence formed by processing the input sequence with an inner convolutional neural network;
generating a weighted intermediate output sequence by weighting the intermediate output sequence with the second weighting sequence; and
superimposing the weighted input sequence and the weighted intermediate output sequence.

13. The system of claim 11 , wherein the convolutional neural network is configured to generate the first weighting sequence or the second weighting sequence based on the intermediate output sequence.

14. The system of claim 13 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the intermediate output sequence.

15. The system of claim 13 , wherein:
generating the first weighting sequence is based on forming a combination of the grid-generation sequence and the intermediate output sequence; and
generating the second weighting sequence is based on forming a combination of the input sequence, the grid-generation sequence, and the intermediate output sequence.

16. The system of claim 13 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence, the grid-generation sequence, and the intermediate output sequence.

17. The system of claim 11 , wherein the convolutional neural network is configured to generate the first weighting sequence and the second weighting sequence without using the intermediate output sequence.

18. The system of claim 17 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the grid-generation sequence.

19. The system of claim 17 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the intermediate grid-generation sequence.

20. A method comprising:
receiving, by a convolutional neural network and from a sensor, an input sequence comprising a plurality of data items captured over time in a scene of a ve

Assignees

• Aptiv Tech Ltd

Inventors

• Zhu Weimeng
• Su Yu
• Nunn Christian

Classifications

• G06V10/62

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/09

  Supervised learning · CPC title