Map creation and localization for autonomous driving applications

Patent metadata

Field	Value
Publication number	US-11698272-B2
Application number	US-202017007873-A
Country	US
Kind code	B2
Filing date	Aug 31, 2020
Priority date	Aug 31, 2019
Publication date	Jul 11, 2023
Grant date	Jul 11, 2023

Abstract

An end-to-end system for data generation, map creation using the generated data, and localization to the created map is disclosed. Mapstreams—or streams of sensor data, perception outputs from deep neural networks (DNNs), and/or relative trajectory data—corresponding to any number of drives by any number of vehicles may be generated and uploaded to the cloud. The mapstreams may be used to generate map data—and ultimately a fused high definition (HD) map—that represents data generated over a plurality of drives. When localizing to the fused HD map, individual localization results may be generated based on comparisons of real-time data from a sensor modality to map data corresponding to the same sensor modality. This process may be repeated for any number of sensor modalities and the results may be fused together to determine a final fused localization result.

First claim

What is claimed is:

1. A method comprising:
causing a vehicle to navigate within an environment based at least on a fused map representative of the environment, wherein the fused map is generated, at least, by:
receiving a plurality of mapstreams corresponding to a plurality of drives;
determining segments of two or more mapstreams of the plurality of mapstreams that are within a threshold distance to each other;
generating, based at least on registering the segments from the two or more mapstreams, a frame graph that includes poses and corresponding pose links associated with the two or more mapstreams;
generating, based at least on dividing the frame graph into road segments, a pose graph that includes a first portion of the poses and a first portion of the pose links corresponding to a first position within a first road segment of the road segments and a second portion of the poses and a second portion of the pose links corresponding to a second position within a second road segment of the road segments; and
generating the fused map based at least on fusing data from the two or more mapstreams according to the pose graph.

2. The method of claim 1 , wherein the fused map is further generated, at least, by:
converting the two or more mapstreams to respective maps,
wherein the fusing the data from the two or more mapstreams comprises fusing the data from the respective maps.

3. The method of claim 2 , wherein an individual map of the respective maps includes two or more map layers corresponding to different sensor modalities.

4. The method of claim 1 , further comprising:
determining, based at least on the fused map, a path for the vehicle through the environment,
wherein the causing the vehicle to navigate through the environment comprises controlling the vehicle to navigate along the path.

5. The method of claim 1 , wherein the plurality of mapstreams are generated by a plurality of vehicles.

6. The method of claim 1 , further comprises receiving, using the vehicle, at least a subset of the fused map corresponding to sensor modalities that the vehicle is equipped with.

7. A method comprising:
receiving data representative of sensor data, perception outputs from one or more deep neural networks (DNNs), and trajectory information corresponding to a plurality of drives;
converting the data to first map data representative of a plurality of maps, an individual map of the plurality of maps corresponding to a drive of the plurality, of drives;
registering, based at least on the trajectory information, a first segment of a first map of the plurality of maps to a second segment of a second map of the plurality of maps;
generating, based at least on the registering, a frame graph that includes poses and corresponding pose links associated with the first segment of the first map and the second segment of the second map;
generating, based at least on assigning at least a portion of the poses and at least a portion of the pose links from the frame graph to road segments, a pose graph;
based at least on the pose graph, generating second map data representative of a fused map corresponding to the first map and the second map, the fused map including the road segments; and
sending data representative of the fused map to a vehicle to cause the vehicle to navigate an environment based at least on the fused map.

8. The method of claim 7 , further comprising localizing the vehicle with respect to a global coordinate system.

9. The method of claim 7 , wherein the individual map includes one or more map layers corresponding to different sensor modalities.

10. The method of claim 7 , further comprising:
receiving additional data corresponding to additional drives based at least on a quality of localization results; and
updating the fused map based at least on the additional data.

11. The method of claim 7 , further comprising generating, based at least on executing one or more optimization algorithms on the pose graph, an updated pose graph, wherein the generating the second map data representative of the fused map is based at least on the updated pose graph.

12. The method of claim 7 , wherein the receiving the data representative of the sensor data, the perception outputs, and the trajectory information comprises receiving pre-processed data representative of the sensor data, the perception outputs, and the trajectory information, the pre-processed data reducing an amount of the data.

13. The method of claim 7 , wherein the data representative of at least one of the sensor data, the perception outputs, or the trajectory information is received in a compressed format, and the method further comprises:
decompressing the data in the compressed format,
wherein the converting the data to the first map data includes converting the data after the decompressing.

14. The method of claim 7 , wherein the registering includes geometric registration.

15. The method of claim 7 , further comprising:
assigning one or more origins to the road segment;
computing one or more transforms between the one or more origins of neighboring road segments of the road segments; and
encoding the one or more transforms in the fused map.

16. The method of claim 7 , further comprising determining to register the first segment and the second segment based at least on global navigation satellite system (GNSS) data corresponding to the first map and the second map.

17. The method of claim 7 , wherein the perception outputs correspond to three-dimensional (3D) representations of detected landmarks, the detected landmarks including one or more of lane dividers, road boundaries, signs, poles, static objects, or vertical structures.

18. The method of claim 7 , wherein the sensor data corresponds to at least one of LiDAR data, RADAR data, or ultrasonic data.

19. A system comprising:
one or more processing units to:
receive a plurality of mapstreams corresponding to a plurality of drives;
determine segments of two or more mapstreams of the plurality of mapstreams that are within a threshold distance to each other;
generate, based at least in part on registering the segments, a frame graph that includes poses and corresponding pose links associated with the segments;
generate, based at least on dividing the frame graph into road segments, a pose graph that includes a first portion of the poses and a first portion of the pose links corresponding to a first road segment of the road segments and a second portion of the poses and a second portion of the pose links corresponding to a second road segment of the road segments;
fuse data from the two or more mapstreams based at least on the pose graph;
generate, based at least on the data from the two or more mapstreams being fused, a fused map; and
send the fused map to a vehicle to cause the vehicle to navigate based at least on the fused map.

20. The system of claim 19 , wherein the one or more processing units are further to:
convert the two or more mapstreams to respective maps,
wherein the data from the two or more mapstreams is fused using the respective maps.

21. The method of claim 20 , wherein an individual respective map of the respective maps includes map layers corresponding to different sensor modalities.

22. The method of claim 19 , wherein to cause the vehicle to navigate based at least on the fused map comprises causing the vehicle to:
determine, based at least on the fused map, a path for the vehicle through an environment; an

Assignees

• Nvidia Corp

Inventors

• Kroepfl Michael
• Akbarzadeh Amir
• Bhargava Ruchi
• Thukral Vaibhav
• Cvijetic Neda
• Cugunovs Vadim
• Nister David
• Henke Birgit
• Eden Ibrahim
• Zhu Youding
• Grabner Michael
• Stojanovic Ivana
• Sheng Yu
• Liu Jeffrey
• Zheng Enliang
• Marr Jordan
• Carley Andrew

Classifications

• G01C21/1652

  with ranging devices, e.g. LIDAR or RADAR · CPC title

• G01C21/3867

  Geometry of map features, e.g. shape points, polygons or for simplified maps · CPC title

• G01C21/3878

  Hierarchical structures, e.g. layering · CPC title

• G01C21/3841 Primary

  Data obtained from two or more sources, e.g. probe vehicles · CPC title

• G06N3/02

  Neural networks · CPC title