Object detection using recurrent neural network and concatenated feature map

Patent metadata

Field	Value
Publication number	US-10452946-B2
Application number	US-201816226142-A
Country	US
Kind code	B2
Filing date	Dec 19, 2018
Priority date	Jan 24, 2017
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

According to one embodiment, a system includes a sensor component and a detection component. The sensor component is configured to obtain a first stream of sensor data and a second stream of sensor data, wherein each of the first stream and second stream comprise a plurality of sensor frames. The detection component is configured to generate a concatenated feature map based on a sensor frame of a first type and a sensor frame of a second type. The detection component is configured to detect one or more objects based on the concatenated feature map. One or more of generating and detecting comprises generating or detecting using a neural network with a recurrent connection that feeds information about features or objects from previous frames.

First claim

The invention claimed is:

1. A method comprising:
extracting object features from a depth map to generate a first feature map by processing the depth map using a neural network, wherein the depth map is based on range sensor data;
extracting object features from an image to generate a second feature map by processing the image using the neural network;
concatenating the first feature map and the second feature map to generate a concatenated feature map; and
detecting one or more objects based on the concatenated map by processing the concatenated map using the neural network.

2. The method of claim 1 , wherein generating the first feature map comprises generating based on high-level features in the depth map, and wherein generating the second feature map comprises generating based on high-level features in the image.

3. The method of claim 2 , wherein generating the concatenated feature map comprises fusing features from the first feature map and the second feature map.

4. The method of claim 1 , wherein the neural network feeds forwards one or more of:
a feature in a previous feature map;
an object detected in a previous frame or time period; or
a feature in a previous depth map.

5. The method of claim 1 , further comprising:
generating a subsequent first feature map based on a subsequent depth map that is based on subsequent range data;
generating a subsequent second feature map based on a subsequent image; and
feeding forward one or more of a feature of the first feature map or the second feature map using recurrent connections in the neural network.

6. The method of claim 1 , wherein the neural network comprises an input layer, one or more hidden layers, and a classification layer, wherein a recurrent connection of the neural network feeds an output of the classification layer from the previous frames into one or more of the input layer or a hidden layer of the one or more hidden layers during generating the feature map or detecting the one or more objects.

7. The method of claim 1 , wherein the concatenated map comprises a fusion of extracted object features from the depth map and extracted object features from the image.

8. The method of claim 1 , wherein the range sensor data comprises one or more of light detection and ranging (LIDAR) sensor data or depth camera data.

9. A system comprising:
a sensor of a vehicle;
a camera of the vehicle; and
a processor that is programmable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
receiving sensor data from the sensor and an image from the camera;
generating a depth map based on the sensor data;
extracting object features from the depth map to generate a first feature map by processing the depth map using a neural network;
extracting object features from an image to generate a second feature map by processing the image using the neural network;
concatenating the first feature map and the second feature map to generate a concatenated feature map; and
detecting one or more objects based on the concatenated map by processing the concatenated map using the neural network.

10. The system of claim 9 , wherein generating the first feature map comprises generated base on high-level features in the depth map, and wherein generating the second feature map comprises generating based on high-level features in the image.

11. The system of claim 10 , wherein generating the concatenated feature map comprises fusing features from the first feature map and the second feature map.

12. The system of claim 9 , wherein the neural network feeds forwards one or more of:
a feature in a previous feature map;
an object detected in a previous frame or time period; or
a feature in a previous depth map.

13. The system of claim 9 , wherein the instructions further comprise:
generating a subsequent first feature map based on a subsequent depth map that is based on subsequent range data;
generating a subsequent second feature map based on a subsequent image; and
feeding forward one or more of a feature of the first feature map or the second feature map using recurrent connections in the neural network.

14. The system of claim 9 , wherein the neural network comprises an input layer, one or more hidden layers, and a classification layer, wherein a recurrent connection of the neural network feeds an output of the classification layer from the previous frames into one or more of the input layer or a hidden layer of the one or more hidden layers during generating the feature map or detecting the one or more objects.

15. Non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:
extract object features from a depth map to generate a first feature map by processing the depth map using a neural network, wherein the depth map is based on range sensor data;
extract object features from an image to generate a second feature map by processing the image using the neural network;
concatenating the first feature map and the second feature map to generate a concatenated feature map; and
detect one or more objects based on the concatenated map by processing the concatenated map using the neural network.

16. The non-transitory computer readable storage media of claim 15 , wherein the instructions cause the one or more processors to generate the first feature map by generating based on high-level features in the depth map and generate the second feature map by generating based on high-level features in the image.

17. The non-transitory computer readable storage media of claim 16 , wherein the instructions cause the one or more processors to generate the concatenated feature map by fusing features from the first feature map and the second feature map.

18. The non-transitory computer readable storage media of claim 15 , wherein the neural network feeds forwards one or more of:
a feature in a previous feature map;
an object detected in a previous frame or time period; or
a feature in a previous depth map.

19. The non-transitory computer readable storage media of claim 15 , wherein the instructions further cause the one or more processors to:
generate a subsequent first feature map based on a subsequent depth map that is based on subsequent range data;
generate a subsequent second feature map based on a subsequent image; and
feed forward one or more of a feature of the first feature map or the second feature map using recurrent connections in the neural network.

20. The non-transitory computer readable storage media of claim 15 , wherein the neural network comprises an input layer, one or more hidden layers, and a classification layer, wherein a recurrent connection of the neural network feeds an output of the classification layer from the previous frames into one or more of the input layer or a hidden layer of the one or more hidden layers during generating the feature map or detecting the one or more objects.

Assignees

• Ford Global Tech Llc

Inventors

• Hotson Guy
• Mahmoudieh Parsa
• Nariyambut Murali Vidya

Classifications

• G01S7/417 Primary

  involving the use of neural networks · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title