Device and a method for extracting dynamic information on a scene using a convolutional neural network

Patent metadata

Field	Value
Publication number	US-2019325241-A1
Application number	US-201916374138-A
Country	US
Kind code	A1
Filing date	Apr 3, 2019
Priority date	Apr 23, 2018
Publication date	Oct 24, 2019
Grant date	—

Abstract

A device for extracting dynamic information comprises a convolutional neural network, wherein the device is configured to receive a sequence of data blocks acquired over time, each of said data blocks comprising a multi-dimensional representation of a scene. The convolutional neural network is configured to receive the sequence as input and to output dynamic information on the scene in response, wherein the convolutional neural network comprises a plurality of modules, and wherein each of said modules is configured to carry out a specific processing task for extracting the dynamic information.

First claim

We claim:

1 . A device for extracting dynamic information comprising:
a convolutional neural network, wherein the device is configured to receive a sequence of data blocks acquired over time, each of said data blocks comprising a multi-dimensional representation of a scene, wherein the convolutional neural network is configured to receive the sequence as input and to output dynamic information on the scene in response, wherein the convolutional neural network comprises a plurality of modules, and wherein each of said modules is configured to carry out a specific processing task for extracting the dynamic information.

2 . The device according to claim 1 ,
wherein a first module is configured to extract image data of the scene from a data block of the sequence, and wherein the image data is formed by a multi-dimensional grid of elements, each of said elements comprising one or more channels.

3 . The device according to claim 1 , wherein
a second module is configured to extract first semantic segmentation data of the scene from image data of the scene, wherein the first semantic segmentation data comprises a classification of the image data for distinguishing between objects and background captured in the image data.

4 . The device according to claim 1 , wherein
a third module is configured to extract second semantic segmentation data of the scene and/or motion data of the scene from first semantic segmentation data of the scene, wherein the first semantic segmentation data comprises a classification of image data of the scene for distinguishing between objects and background captured in the image data, and wherein the motion data represents the motion of objects captured in the image data, and wherein the third module is configured to extract the second semantic segmentation data and/or motion data on the basis of the first semantic segmentation data captured at a plurality of different time instances.

5 . The device according to claim 4 , wherein
the third module is formed by a recurrent neural network.

6 . The device according to claim 4 , wherein
a fourth module is configured to extract object data from the second semantic segmentation data and the motion data, wherein the object data represents a spatial occupancy of objects in the scene, wherein the object data additionally represents the velocity of objects in the scene.

7 . The device according to claim 6 , wherein
for a given object in the scene, the object data comprises a bounding box around the object, in particular wherein the object data additionally comprises the velocity of the object.

8 . The device according to at least one of claim 4 , wherein
a fifth module is configured to extract free-space data from the second semantic segmentation data and the motion data, wherein the free-space data represents the spatial occupancy of free space in the scene.

9 . The device according to claim 8 , wherein the dynamic information comprises the object data, the free-space data or the motion data.

10 . A system for processing data sequences, the system comprising a sensor for capturing a data sequence and a device according to one of the preceding claims.

11 . The system according to claim 10 , wherein
the sensor comprises at least one of a radar sensor, a light detection and ranging sensor, an ultrasonic sensor or a camera and wherein the data sequence represents data acquired by means of the sensor.

12 . A vehicle with a system according to claim 10 , wherein
a control unit of the vehicle is configured to receive dynamic information on the surrounding of the vehicle extracted by means of the device, and the control unit of the vehicle is further configured to control the vehicle with respect to the extracted information or to output a warning signal if the information meets a predetermined condition.

13 . The vehicle according to claim 12 ,
wherein the dynamic information represents the position and the movement of objects in the surrounding of the vehicle.

14 . A method of extracting dynamic information on a scene, the method comprising:
acquiring a sequence of data blocks using at least one sensor, each of said data blocks comprising a multi-dimensional representation of a scene, extracting dynamic information on the scene by using a convolutional neural network, wherein the convolutional neural network is configured to receive the data blocks as input and to output the dynamic information in response, wherein the convolutional neural network comprises a plurality of modules, and wherein each of said modules is configured to carry out a specific processing task for extracting the dynamic information.

Assignees

• Aptiv Tech Ltd

Inventors

• Nunn Christian
• Zhu Weimeng
• Su Yu

Classifications

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title