Generation and update of HD maps using data from heterogeneous sources

Patent metadata

Field	Value
Publication number	US-11537868-B2
Application number	US-201715811489-A
Country	US
Kind code	B2
Filing date	Nov 13, 2017
Priority date	Nov 13, 2017
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

A method includes a computing system accessing a training sample that includes first sensor data obtained using a first sensor at a first geographic location, and first metadata comprising information relating to the first sensor. The system may train a machine-learning model by generating first map data by processing the training sample using the model and updating the model based on the generated first map data and target map data associated with the first geographic location. The system may then access second sensor data and second metadata, where the second sensor data is obtained using a second sensor. The system may generate second map data associated with a second geographic location by processing the second sensor data and the second metadata using the trained model. A high-definition map may be generated using the second map data.

First claim

What is claimed is:

1. A method comprising, by a computing device:
accessing a plurality of training samples from a training data set, wherein the plurality of training samples comprises at least (1) a first sensor data sample including first sensor data, generated using a first sensor, a first geographic location and first contextual information relating to the first sensor data as generated by the first sensor, and (2) a second sensor data sample including second sensor data, generated using a second sensor, of the first geographic location and second contextual information relating to the second sensor data as generated by the second sensor, wherein the first sensor is different from the second sensor, and wherein each of the first sensor data sample and the second sensor data sample comprises a known representation of the first geographic location;
accessing target map data associated with the first geographic location, wherein the target map data is based on an existing high-definition (HD) map;
subsequent to accessing the plurality of training samples and accessing the target map data, training a trainable model by:
generating first map data by using the trainable model to encode the first sensor data sample and the second sensor data sample into a first latent representation in a common data space; and
updating the trainable model by comparing the generated first map data and the target map data; and

subsequent to determining that the training of the trainable model is complete:
generating an updated HD map by utilizing the trained trainable model based at least in part on the existing HD map by:
accessing a third sensor data sample including third sensor data, generated using a third sensor, of a second geographic location and third contextual information relating to the third sensor data as generated by the third sensor; and
generating second map data associated with the second geographic location by using the trained trainable model to encode the third sensor data and the third contextual information into a second latent representation in the common data space.

2. The method of claim 1 , wherein the trainable model comprises a convolutional neural network and a deconvolutional neural network, wherein an output of the convolutional neural network is configured to be an input of the deconvolutional neural network.

3. The method of claim 2 , wherein generating the first map data further comprises:
using the convolutional neural network to encode the first sensor data sample and the second sensor data sample into the first latent representation in the common data space; and
using the deconvolutional neural network to decode the first latent representation.

4. The method of claim 2 , wherein:
the trainable model comprises a second convolutional neural network;
an output of the second convolution neural network is configured to be a second input of the deconvolutional neural network; and
wherein generating the first map data further comprises:
generating the first latent representation of the plurality of training samples by (1) processing the first sensor data sample using the convolutional neural network and (2) processing the second sensor data sample using the second convolutional neural network; and
generating the first map data by processing the first latent representation using the deconvolutional neural network.

5. The method of claim 1 , wherein the plurality of training samples further comprises a fourth sensor data sample including fourth sensor data, generated using a fourth sensor, of the first geographic location and fourth contextual information relating to the fourth sensor data as generated by the fourth sensor, the first sensor, the second sensor, and the fourth sensor having different sensor types.

6. The method of claim 1 , wherein the first contextual information comprises at least one of a spatial location of the first sensor relative to a reference point, an orientation of the first sensor, a capability of the first sensor, a configuration of the first sensor, a model of the first sensor, or a type of the first sensor.

7. The method of claim 1 , wherein the plurality of training samples further comprises first environmental data associated with a first time at which the first sensor data were generated using the first sensor and second environmental data associated with a second time at which the second sensor data were generated using the second sensor, each of the first environmental data and the second environmental data comprising at least one of wind speed, temperature, precipitation, time of day, lighting condition, or visibility condition.

8. The method of claim 1 , wherein generating the first map data further comprises processing an object classification utilizing the trained trainable model, wherein the object classification identifies a type of object detected in each of the first sensor data and the second sensor data.

9. The method of claim 1 , further comprising:
transmitting the updated HD map to one or more vehicles, wherein the updated HD map is configured to be used by the one or more vehicles for driving.

10. The method of claim 1 , wherein generating the updated HD map comprises utilizing the trained trainable model to generate an updated portion of the updated HD map, the updated portion of the updated HD map corresponding to the second geographical location.

11. The method of claim 1 , wherein training the trainable model further comprises updating the trainable model based on a comparison of the generated second map data and the target map data.

12. The method of claim 1 , wherein determining that the training of the trainable model is complete comprises:
determining a score based on the comparison of the generated first map data and the target map data; and
comparing the score to a threshold requirement associated with the target map data.

13. The method of claim 1 , wherein encoding the first sensor data sample and the second sensor data sample into the first latent representation in the common data space reduces discrepancies between the first sensor data as generated by the first sensor and the second sensor data as generated by the second sensor.

14. A system, comprising:
one or more processors and one or more computer-readable non-transitory storage media in communication with the one or more processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by the one or more processors to cause the system to perform operations comprising:
accessing a plurality of training samples from a training data set, wherein the plurality of training samples comprises at least (1) a first sensor data sample including first sensor data, generated using a first sensor, of a first geographic location and first contextual information relating to the first sensor data as generated by the first sensor, and (2) a second sensor data sample including second sensor data, generated using a second sensor, of the first geographic location and second contextual information relating to the second sensor data as generated by the second sensor, wherein the first sensor is different from the second sensor, and wherein each of the first sensor data sample and the second sensor data sample comprises a known representation of the first geographic location;
accessing target map data associated with the first geographic location, wherein the target map data is based on an existing high-definition (HD) map;
subsequent to accessing the plurality of training samples and accessing the target map data, training a trainable model by:
generating

Assignees

• Lyft Inc

Inventors

• Arditi Gil

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N5/025

  Extracting rules from data · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G01C21/3878

  Hierarchical structures, e.g. layering · CPC title