Learning method and learning device for integrating image acquired by camera and point-cloud map acquired by radar or LiDAR corresponding to image at each of convolution stages in neural network and testing method and testing device using the same

Patent metadata

Field	Value
Publication number	US-10408939-B1
Application number	US-201916262984-A
Country	US
Kind code	B1
Filing date	Jan 31, 2019
Priority date	Jan 31, 2019
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

A method for integrating, at each convolution stage in a neural network, an image generated by a camera and its corresponding point-cloud map generated by a radar, a LiDAR, or a heterogeneous sensor fusion is provided to be used for an HD map update. The method includes steps of: a computing device instructing an initial operation layer to integrate the image and its corresponding original point-cloud map, to generate a first fused feature map and a first fused point-cloud map; instructing a transformation layer to apply a first transformation operation to the first fused feature map, and to apply a second transformation operation to the first fused point-cloud map; and instructing an integration layer to integrate feature maps outputted from the transformation layer, to generate a second fused point-cloud map. By the method, an object detection and a segmentation can be performed more efficiently with a distance estimation.

First claim

What is claimed is:

1. A method for integrating, at each convolution stage in a neural network, at least one image generated by at least one camera and its corresponding at least one point-cloud map generated by at least one radar or at least one LiDAR, comprising steps of:
(a) a computing device instructing at least one initial operation layer to integrate at least one original image generated by the camera and its corresponding at least one original point-cloud map generated by the radar or the LiDAR, to thereby generate (i) at least one first fused feature map by adding depth information included in the original point-cloud map to the original image and (ii) at least one first fused point-cloud map by adding color information included in the original image to the original point-cloud map;
(b) the computing device instructing at least one transformation layer to generate a (1_1)-st intermediate feature map by applying at least one first transformation operation to the first fused feature map, and to generate a (1_2)-nd intermediate feature map by applying at least one second transformation operation to the first fused point-cloud map; and
(c) the computing device instructing at least one integration layer to generate a second fused feature map by integrating the (1_1)-st intermediate feature map and the (1_2)-nd intermediate feature map, and to generate a second fused point-cloud map by applying at least one mapping operation to the second fused feature map.

2. The method of claim 1 , wherein the method further comprises a step of:
(d) the computing device, as a result of repeating the steps of (b) and (c), (i) instructing the transformation layer to generate an (N_1)-st intermediate feature map by applying the first transformation operation to an N-th fused feature map created by the integration layer, to generate an (N_2)-nd intermediate feature map by applying the second transformation operation to an N-th fused point-cloud map created by the integration layer, and (ii) instructing the integration layer to generate an (N+1)-th fused feature map by integrating the (N_1)-st intermediate feature map and the (N_2)-nd intermediate feature map, and to generate an (N+1)-th fused point-cloud map by applying the mapping operation to the (N+1)-th fused feature map.

3. The method of claim 2 , wherein the method further comprises a step of:
(e) the computing device instructing at least one output layer to perform at least part of operations required for autonomous driving which include at least part of an object detection, a semantic segmentation and a depth estimation, by referring to at least part of the (N+1)-th fused feature map and the (N+1)-th fused point-cloud map.

4. The method of claim 3 , wherein the method further comprises a step of:
(f) the computing device, if at least one output of the neural network created by the output layer is generated, learning at least part of one or more parameters of the neural network by referring to the output and its at least one corresponding GT.

5. The method of claim 1 , wherein, at the step of (a), the first fused feature map includes (i) original color information, on each pixel, in the original image, and (ii) the depth information on the each pixel, generated by referring to original coordinate information on each position in a three dimensional space near the radar or the LiDAR wherein the each position is included in the original point-cloud map, and wherein the first fused point-cloud map includes (i) the original coordinate information and (ii) the color information on the each position acquired by referring to the original color information.

6. The method of claim 1 , wherein, at the step of (b), the (1_1)-st intermediate feature map is generated by applying the first transformation operation including at least one convolution operation to the first fused feature map.

7. The method of claim 6 , wherein, at the step of (b), the (1_1)-st intermediate feature map is generated by applying the first transformation operation further including at least one ReLU operation and at least one pooling operation to the first fused feature map.

8. The method of claim 1 , wherein, at the step of (b), the (1_2)-nd intermediate feature map is generated by applying the second transformation operation including at least one neural network operation, at least one inverse mapping operation, and at least one convolution operation to the first fused point-cloud map, and
wherein the inverse mapping operation correlates (i) the depth information, included in the first fused point-cloud map, in a form of three dimensional coordinates linked with the color information with (ii) each of features in the (1_1)-st intermediate feature map.

9. The method of claim 1 , wherein, at the step of (c), the second fused feature map is generated by concatenating the (1_1)-st intermediate feature map and the (1_2)-nd intermediate feature map in a direction of a channel.

10. The method of claim 1 , wherein, at the step of (c), the mapping operation correlates (i) each of feature values in the second fused feature map with (ii) each position in a three dimensional space near the radar or the LiDAR.

11. A method for testing and using integration of, at each convolution stage in a neural network, at least one image generated by at least one camera and its corresponding at least one point-cloud map generated by at least one radar or at least one LiDAR, comprising steps of:
(a) a testing device, on condition that (1) a learning device has performed processes of instructing at least one initial operation layer to integrate at least one original training image generated by the camera and its corresponding at least one original point-cloud map for training generated by the radar or the LiDAR, to thereby generate (i) at least one first fused feature map for training by adding depth information for training included in the original point-cloud map for training to the original training image and (ii) at least one first fused point-cloud map for training by adding color information for training included in the original training image to the original point-cloud map for training, (2) the learning device has performed processes of instructing at least one transformation layer to generate a (1_1)-st intermediate feature map for training by applying at least one first transformation operation to the first fused feature map for training, and to generate a (1_2)-nd intermediate feature map for training by applying at least one second transformation operation to the first fused point-cloud map for training, (3) the learning device has performed processes of instructing at least one integration layer to generate a second fused feature map for training by integrating the (1_1)-st intermediate feature map for training and the (1_2)-nd intermediate feature map for training, and to generate a second fused point-cloud map for training by applying at least one mapping operation to the second fused feature map for training, (4) the learning device, as a result of repeating the steps of (2) and (3), has performed processes of (i) instructing the transformation layer to generate an (N_1)-st intermediate feature map for training by applying the first transformation operation to an N-th fused feature map for training created by the integration layer, to generate an (N_2)-nd intermediate feature map for training by applying the second transformation operation to an N-th fused point-cloud map for training created by the integration layer, (ii) instructing the integration layer to generate an (N+1)-th fused feature map for training by integrating the (N_1)-st intermediate feature map for training and the (N_2)-nd intermediate feature map

Assignees

• Stradvision Inc

Inventors

• Kim Kye-Hyeon
• Kim Yongjoong
• Kim Insu
• Kim Hak-Kyoung
• Nam Woonhyun
• Boo Sukhoon
• Sung Myungchul
• Yeo Donghun
• Ryu Wooju
• Jang Taewoong
• Jeong Kyungjoong
• Je Hongmo
• Cho Hojin

Classifications

• G01S13/89

  for mapping or imaging · CPC title

• G01S13/931

  of land vehicles · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title

• G01S17/931

  of land vehicles · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title