Object detection using recurrent neural network and concatenated feature map

Patent metadata

Field	Value
Publication number	US-11062167-B2
Application number	US-201916576277-A
Country	US
Kind code	B2
Filing date	Sep 19, 2019
Priority date	Jan 24, 2017
Publication date	Jul 13, 2021
Grant date	Jul 13, 2021

Abstract

According to one embodiment, a system includes a sensor component and a detection component. The sensor component is configured to obtain a first stream of sensor data and a second stream of sensor data, wherein each of the first stream and second stream comprise a plurality of sensor frames. The detection component is configured to generate a concatenated feature map based on a sensor frame of a first type and a sensor frame of a second type. The detection component is configured to detect one or more objects based on the concatenated feature map. One or more of generating and detecting comprises generating or detecting using a neural network with a recurrent connection that feeds information about features or objects from previous frames.

First claim

The invention claimed is:

1. A method comprising:
generating a first feature map based on image data;
generating a second feature map based on depth map data;
fusing the first feature map and the second feature map to generate a fused feature map; and
detecting one or more objects based on the fused feature map by processing the fused feature map using a neural network.

2. The method of claim 1 , wherein fusing the first feature map and the second feature map comprises concatenating the first feature map and the second feature map.

3. The method of claim 1 , wherein:
generating the first feature map comprises extracting object features from the image data by processing the image data using the neural network; and
generating the second feature map comprises extracting object features from the depth map data by processing the depth map using the neural network, wherein the depth map is based on range sensor data.

4. The method of claim 1 , further comprising:
generating a subsequent first feature map based on subsequent image data captured after the image data;
generating a subsequent second feature map based on subsequent depth map data captured by a range sensor after the depth map data; and
feeding forward one or more of a feature of the first feature map or the second feature map using recurrent connection in the neural network.

5. The method of claim 1 , wherein the neural network comprises an input layer, one or more hidden layers, and a classification layer.

6. The method of claim 5 , wherein a recurrent connection of the neural network feeds an output of the classification layer from previous frames into one or more of the input layer or a hidden layer of the one or more hidden layers during detecting the one or more objects based on the fused feature map.

7. The method of claim 1 , wherein the fused feature map comprises a concatenated feature map fusing extracted object features from the image data with extracted object features from the depth map data.

8. The method of claim 1 , wherein the depth map data comprises one or more of data received from a light detection and ranging (LIDAR) sensor or a depth camera.

9. A system comprising:
a range sensor of a vehicle;
a camera of a vehicle; and
a processor that is programmable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
generating a first feature map based on image data;
generating a second feature map based on depth map data;
fusing the first feature map and the second feature map to generate a fused feature map; and
detecting one or more objects based on the fused feature map by processing the fused feature map using a neural network.

10. The system of claim 9 , wherein the instructions are such that fusing the first feature map and the second feature map comprises concatenating the first feature map and the second feature map.

11. The system of claim 9 , wherein the instructions are such that:
generating the first feature map comprises extracting object features from the image data by processing the image data using the neural network; and
generating the second feature map comprises extracting object features from the depth map data by processing the depth map using the neural network, wherein the depth map is based on range sensor data.

12. The system of claim 9 , wherein the instructions further comprise:
generating a subsequent first feature map based on subsequent image data captured after the image data;
generating a subsequent second feature map based on subsequent depth map data captured by a range sensor after the depth map data; and
feeding forward one or more of a feature of the first feature map or the second feature map using recurrent connection in the neural network.

13. The system of claim 9 , further comprising the neural network, wherein the neural network comprises an input layer, one or more hidden layers, and a classification layer, and wherein a recurrent connection of the neural network feeds an output of the classification layer from previous frames into one or more of the input layer or a hidden layer of the one or more hidden layers during detecting the one or more objects based on the fused feature map.

14. The system of claim 9 , wherein the instructions are such that the fused feature map comprises a concatenated feature map fusing extracted object features from the image data with extracted object features from the depth map data.

15. Non-transitory computer readable storage media storing instructions that, when executed by one or more processors cause the one or more processors to execute the following:
generating a first feature map based on image data;
generating a second feature map based on depth map data;
fusing the first feature map and the second feature map to generate a fused feature map; and
detecting one or more objects based on the fused feature map by processing the fused feature map using a neural network.

16. The non-transitory computer readable storage media of claim 15 , wherein the instructions are such that fusing the first feature map and the second feature map comprises concatenating the first feature map and the second feature map.

17. The non-transitory computer readable storage media of claim 15 , wherein the instructions are such that:
generating the first feature map comprises extracting object features from the image data by processing the image data using the neural network; and
generating the second feature map comprises extracting object features from the depth map data by processing the depth map using the neural network, wherein the depth map is based on range sensor data.

18. The non-transitory computer readable storage media of claim 15 , wherein the instructions further comprise:
generating a subsequent first feature map based on subsequent image data captured after the image data;
generating a subsequent second feature map based on subsequent depth map data captured by a range sensor after the depth map data; and
feeding forward one or more of a feature of the first feature map or the second feature map using recurrent connection in the neural network.

19. The non-transitory computer readable storage media of claim 15 , wherein the neural network comprises an input layer, one or more hidden layers, and a classification layer, and wherein a recurrent connection of the neural network feeds an output of the classification layer from previous frames into one or more of the input layer or a hidden layer of the one or more hidden layers during detecting the one or more objects based on the fused feature map.

20. The non-transitory computer readable storage media of claim 15 , wherein the instructions are such that the fused feature map comprises a concatenated feature map fusing extracted object features from the image data with extracted object features from the depth map data.

Assignees

• Ford Global Tech Llc

Inventors

• Hotson Guy
• Mahmoudieh Parsa
• Nariyambut Murali Vidya

Classifications

• G01S7/417 Primary

  involving the use of neural networks · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title