Lidar localization using RNN and LSTM for temporal smoothness in autonomous driving vehicles

Patent metadata

Field	Value
Publication number	US-11364931-B2
Application number	US-201916337391-A
Country	US
Kind code	B2
Filing date	Jan 30, 2019
Priority date	Jan 30, 2019
Publication date	Jun 21, 2022
Grant date	Jun 21, 2022

Abstract

In one embodiment, a method for temporal smoothness in localization results for an autonomous driving vehicle includes: creating a probability offset volume that represents an overall matching cost between a first set of keypoints from the online point cloud and a second set of keypoints from a pre-built point cloud map for each of a series of sequential light detection and ranging (LiDAR) frames in an online point cloud. The method also includes compressing the probability offset volume into multiple probability vectors across a X dimension, a Y dimension and a yaw dimension; providing each probability vector of the probability offset volume to a number of recurrent neural networks (RNNs); and generating, by the RNNs, a trajectory of location results across the plurality of sequential LiDAR frames.

First claim

What is claimed is:

1. A computer-implemented method for an autonomous driving vehicle (ADV), the method comprising:
creating, for each of a plurality of sequential light detection and ranging (LiDAR) frames in an online point cloud, a probability offset volume that represents an overall matching cost between a first set of keypoints from the online point cloud and a second set of keypoints from a pre-built point cloud map;
compressing the probability offset volume into a plurality of probability vectors across a X dimension, a Y dimension, and a yaw dimension;
providing each probability vector of the probability offset volume to a plurality of recurrent neural networks (RNNs); and
generating, by the plurality of RNNs, a trajectory of localization results across the plurality of sequential LiDAR frames.

2. The method of claim 1 , wherein the probability offset volume includes a plurality of sub volumes, wherein each sub volume represents an overall matching cost between the online point cloud and the pre-built point cloud map for a particular location offset of the ADV.

3. The method of claim 1 , wherein each of the RNNs includes a plurality of long short term memory (LSTM) units.

4. The method of claim 3 , wherein each of the probability vectors is provided as an input to one of the plurality of LSTM units.

5. The method of claim 1 , wherein the plurality of RNNs smoothen the trajectory of location results across the plurality of sequential LiDAR frames based on learned historical information from the plurality of sequential LiDAR frames.

6. The method of claim 1 , wherein the ADV is to have a plurality of predicted poses across the plurality of sequential LiDAR frames.

7. The method of claim 1 , wherein the probability offset volume is based on a cost volume constructed from a first set of feature descriptors extracted the online point cloud and a second of feature descriptors extracted from the pre-built point cloud map.

8. A system for an autonomous driving vehicle (ADV), the system comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
creating, for each of a plurality of sequential light detection and ranging (LiDAR) frames in an online point cloud, a probability offset volume that represents an overall matching cost between a first set of keypoints from the online point cloud and a second set of keypoints from a pre-built point cloud map,
compressing the probability offset volume into a plurality of probability vectors across a X dimension, a Y dimension and a yaw dimension,
providing each probability vector of the probability offset volume to a plurality of recurrent neural networks (RNNs), and
generating, by the plurality of RNNs, a trajectory of localization results across the plurality of sequential LiDAR frames.

9. The system of claim 8 , wherein the probability offset volume includes a plurality of sub volumes, wherein each sub volume represents an overall matching cost between the online point cloud and the pre-built point cloud map for a particular location offset of the ADV.

10. The system of claim 8 , wherein each of the RNNs includes a plurality of long short term memory (LSTM) units.

11. The system of claim 10 , wherein each of the probability vectors is provided as an input to one of the plurality of LSTM units.

12. The system of claim 8 , wherein the plurality of RNNs smoothen the trajectory of location results across the plurality of sequential LiDAR frames based on learned historical information from the plurality of sequential LiDAR frames.

13. The system of claim 8 , wherein the ADV is to have a plurality of predicted poses across the plurality of sequential LiDAR frames.

14. The system of claim 8 , wherein the probability offset volume is based on a cost volume constructed from a first set of feature descriptors extracted the online point cloud and a second of feature descriptors extracted from the pre-built point cloud map.

15. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for an autonomous driving vehicle (ADV), the operations comprising:
creating, for each of a plurality of sequential light detection and ranging (LiDAR) frames in an online point cloud, a probability offset volume that represents an overall matching cost between a first set of keypoints from the online point cloud and a second set of keypoints from a pre-built point cloud map;
compressing the probability offset volume into a plurality of probability vectors across a X dimension, a Y dimension and a yaw dimension;
providing each probability vector of the probability offset volume to a plurality of recurrent neural networks (RNNs); and
generating, by the plurality of RNNs, a trajectory of localization results across the plurality of sequential LiDAR frames.

16. The machine-readable medium of claim 15 , wherein the probability offset volume includes a plurality of sub volumes, wherein each sub volume represents an overall matching cost between the online point cloud and the pre-built point cloud map for a particular location offset of the ADV.

17. The machine-readable medium of claim 15 , wherein each of the RNNs includes a plurality of long short term memory (LSTM) units.

18. The machine-readable medium of claim 17 , wherein each of the probability vectors is provided as an input to one of the plurality of LSTM units.

19. The machine-readable medium of claim 15 , wherein the plurality of RNNs smoothen the trajectory of location results across the plurality of sequential LiDAR frames based on learned historical information from the plurality of sequential LiDAR frames.

20. The machine-readable medium of claim 15 , wherein the ADV is to have a plurality of predicted poses across the plurality of sequential LiDAR frames.

21. The machine-readable medium of claim 15 , wherein the probability offset volume is based on a cost volume constructed from a first set of feature descriptors extracted the online point cloud and a second of feature descriptors extracted from the pre-built point cloud map.

Assignees

• Baidu Usa Llc
• Baidu Com Times Tech Beijing Co Ltd

Inventors

• Lu Weixin
• Zhou Yao
• Wan Guowei
• Hou Shenhua
• Song Shiyu

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title