Device and a Method for Processing Data Sequences Using a Convolutional Neural Network

Patent metadata

Field	Value
Publication number	US-2023104196-A1
Application number	US-202218047680-A
Country	US
Kind code	A1
Filing date	Oct 19, 2022
Priority date	Apr 23, 2018
Publication date	Apr 6, 2023
Grant date	—

Abstract

A device for processing data sequences by means of a convolutional neural network is configured to carry out the following steps: receiving an input sequence comprising a plurality of data items captured over time using a sensor, each of said data items comprising a multi-dimensional representation of a scene, generating an output sequence representing the input sequence processed item-wise by the convolutional neural network, wherein generating the output sequence comprises: generating a grid-generation sequence based on a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the output sequence or the grid-generation sequence, generating a sampling grid on the basis of the grid-generation sequence, generating an intermediate output sequence by sampling from the past portion of the output sequence according to the sampling grid, and generating the output sequence based on a weighted combination of the intermediate output sequence and the input sequence.

First claim

What is claimed is:

1 . A device comprising:
a convolutional neural network configured to:
receive, from a sensor, an input sequence comprising a plurality of data items captured over time in a scene of a vehicle;
generate an output sequence representing the input sequence processed item-wise by the convolutional neural network by at least:
generating an intermediate output sequence by sampling from a past portion of the output sequence according to a sampling grid;
generating, based on a grid-generation sequence, the sampling grid item-wise, the grid-generation sequence being a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the grid-generation sequence;
generating, based on the intermediate output sequence, grid-generation sequence, the intermediate grid-generation sequence, or the input sequence, a first weighting sequence and a second weighting sequence;
generating, based on the first weighted sequence and the second weighted sequence, a weighted combination of the intermediate output sequence and the input sequence; and
generating, based on the weighted combination of the intermediate output sequence and the input sequence, the output sequence.

2 . The device of claim 1 , wherein the convolutional neural network is further configured to generate the output sequence by:
generating a weighted input sequence by weighting an intermediate input sequence with the first weighting sequence, the intermediate input sequence formed by processing the input sequence with an inner convolutional neural network; generating a weighted intermediate output sequence by weighting the intermediate output sequence with the second weighting sequence; and superimposing the weighted input sequence and the weighted intermediate output sequence to generate the weighted combination of the intermediate output sequence and the input sequence.

3 . The device of claim 2 , wherein the convolutional neural network is configured to generate the first weighting sequence or the second weighting sequence based on the intermediate output sequence.

4 . The device of claim 3 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the intermediate output sequence.

5 . The device of claim 3 , wherein:
generating the first weighting sequence is based on forming a combination of the grid-generation sequence and the intermediate output sequence; and generating the second weighting sequence is based forming a combination of on the input sequence, the grid-generation sequence, and the intermediate output sequence.

6 . The device of claim 3 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence, the grid-generation sequence, and the intermediate output sequence.

7 . The device of claim 2 , wherein the convolutional neural network is configured to generate the first weighting sequence and the second weighting sequence without using the intermediate output sequence.

8 . The device of claim 7 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the grid-generation sequence.

9 . The device of claim 7 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the intermediate grid-generation sequence.

10 . The device of claim 7 , wherein the convolutional neural network is further configured to generate the grid-generation sequence by:
processing the superimposed weighted input sequence and the weighted intermediate output sequence with an inner convolutional neural network.

11 . A system comprising:
at least one sensor for capturing a data sequence; and a device comprising:
a convolutional neural network configured to:
receive, from the at least one sensor, an input sequence comprising a plurality of data items captured over time in a scene of a vehicle;
generate an output sequence representing the input sequence processed item-wise by the convolutional neural network by at least:
generating an intermediate output sequence by sampling from a past portion of the output sequence according to a sampling grid;
generating, based on a grid-generation sequence, the sampling grid item-wise, the grid-generation sequence being a combination of the input sequence and an intermediate grid-generation sequence representing a past portion of the grid-generation sequence;
generating, based on the intermediate output sequence, grid-generation sequence, the intermediate grid-generation sequence, or the input sequence, a first weighting sequence and a second weighting sequence;
generating, based on the first weighted sequence and the second weighted sequence, a weighted combination of the intermediate output sequence and the input sequence; and
generating, based on the weighted combination of the intermediate output sequence and the input sequence, the output sequence.

12 . The system of claim 11 , wherein the convolutional neural network is further configured to generate the output sequence by:
generating a weighted input sequence by weighting an intermediate input sequence with the first weighting sequence, the intermediate input sequence formed by processing the input sequence with an inner convolutional neural network; generating a weighted intermediate output sequence by weighting the intermediate output sequence with the second weighting sequence; and superimposing the weighted input sequence and the weighted intermediate output sequence.

13 . The system of claim 11 , wherein the convolutional neural network is configured to generate the first weighting sequence or the second weighting sequence based on the intermediate output sequence.

14 . The system of claim 13 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the intermediate output sequence.

15 . The system of claim 13 , wherein:
generating the first weighting sequence is based on forming a combination of the grid-generation sequence and the intermediate output sequence; and generating the second weighting sequence is based forming a combination of on the input sequence, the grid-generation sequence, and the intermediate output sequence.

16 . The system of claim 13 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence, the grid-generation sequence, and the intermediate output sequence.

17 . The system of claim 11 , wherein the convolutional neural network is configured to generate the first weighting sequence and the second weighting sequence without using the intermediate output sequence.

18 . The system of claim 17 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the grid-generation sequence.

19 . The system of claim 17 , wherein:
generating the first weighting sequence and the second weighting sequence is based on forming a combination of the input sequence and the intermediate grid-generation sequence.

20 . A method comprising:
receiving, by a convolutional neural network and from a sensor, an input sequence comprising a plurality of d

Assignees

• Aptiv Tech Ltd

Inventors

• Zhu Weimeng
• Su Yu
• Nunn Christian

Classifications

• G06V10/62

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/09

  Supervised learning · CPC title