Generating training data for deep learning models for building high definition maps

Patent metadata

Field	Value
Publication number	US-2021004021-A1
Application number	US-202016919131-A
Country	US
Kind code	A1
Filing date	Jul 2, 2020
Priority date	Jul 5, 2019
Publication date	Jan 7, 2021
Grant date	—

Abstract

According to an aspect of an embodiment, operations may comprise receiving sensor data from one or more vehicles, determining, by combining the received sensor data, a high definition map comprising a point cloud, and labeling one or more objects in the point cloud. The operations may also comprise generating training data by receiving a new image captured by one of the vehicles, receiving a pose of the vehicle when the new image was captured, determining an object having a label in the point cloud that is observable from the pose of the vehicle, determining a position of the object in the new image, and labeling the new image by assigning the label of the object to the new image, the labeled new image comprising the training data. The operations may also comprise training a deep learning model using the training data.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
receiving sensor data from one or more vehicles; determining, by combining the received sensor data, a high definition map comprising a point cloud; labeling one or more objects in the point cloud, each of the one or more objects represented as a set of points from the point cloud; generating training data by:
receiving a new image captured by one of the vehicles,
receiving a pose of the vehicle when the new image was captured,
determining an object in the point cloud that is observable from the pose of the vehicle, the object having a label,
determining a position of the object in the new image, and
labeling the new image by assigning the label of the object to the new image; the labeled new image comprising the training data; and
training a deep learning model using the training data.

2 . The computer-implemented method of claim 1 , wherein the labeling of the one or more objects comprises:
receiving an image; labeling an object in the image by recognizing the object using a trained deep learning model; identifying the set of points in the point cloud corresponding to the object; and labeling the set of points based on the label of the object.

3 . The computer-implemented method of claim 1 , wherein the set of points is identified using a bounding box.

4 . The computer-implemented method of claim 1 , wherein the labeling of the one or more objects further comprises generating training labels by propagating map features to labels.

5 . The computer-implemented method of claim 4 , wherein the labeling of the one or more objects further comprises selecting labels to review by applying filters to the labels and setting a sampling rate.

6 . The computer-implemented method of claim 5 , wherein the labeling of the one or more objects further comprises reviewing labels by verifying accuracy of labels.

7 . The computer-implemented method of claim 5 , wherein the labeling of the one or more objects further comprises creating a dataset by selecting a subset of labels for the training of the deep learning model.

8 . One or more non-transitory computer-readable storage media storing instructions that in response to being executed by one or more processors, cause a computer system to perform operations, the operations comprising:
receiving sensor data from one or more vehicles; determining, by combining the received sensor data, a high definition map comprising a point cloud; labeling one or more objects in the point cloud, each of the one or more objects represented as a set of points from the point cloud; generating training data by:
receiving a new image captured by one of the vehicles,
receiving a pose of the vehicle when the new image was captured,
determining an object in the point cloud that is observable from the pose of the vehicle, the object having a label,
determining a position of the object in the new image, and
labeling the new image by assigning the label of the object to the new image; the labeled new image comprising the training data; and
training a deep learning model using the training data.

9 . The one or more non-transitory computer-readable storage media of claim 8 , wherein the labeling of the one or more objects comprises:
receiving an image; labeling an object in the image by recognizing the object using a trained deep learning model; identifying the set of points in the point cloud corresponding to the object; and labeling the set of points based on the label of the object.

10 . The one or more non-transitory computer-readable storage media of claim 8 , wherein the set of points is identified using a bounding box.

11 . The one or more non-transitory computer-readable storage media of claim 8 , wherein the labeling of the one or more objects further comprises generating training labels by propagating map features to labels.

12 . The one or more non-transitory computer-readable storage media of claim 8 , wherein the labeling of the one or more objects further comprises selecting labels to review by applying filters to the labels and setting a sampling rate.

13 . The one or more non-transitory computer-readable storage media of claim 8 , wherein the labeling of the one or more objects further comprises reviewing labels by verifying accuracy of labels.

14 . The one or more non-transitory computer-readable storage media of claim 8 , wherein the labeling of the one or more objects further comprises creating a dataset by selecting a subset of labels for the training of the deep learning model.

15 . A computer system comprising:
one or more processors; and one or more non-transitory computer readable media storing instructions that in response to being executed by the one or more processors, cause the computer system to perform operations, the operations comprising:
receiving sensor data from one or more vehicles;
determining, by combining the received sensor data, a high definition map comprising a point cloud;
labeling one or more objects in the point cloud, each of the one or more objects represented as a set of points from the point cloud;
generating training data by:
receiving a new image captured by one of the vehicles,
receiving a pose of the vehicle when the new image was captured,
determining an object in the point cloud that is observable from the pose of the vehicle, the object having a label,
determining a position of the object in the new image, and
labeling the new image by assigning the label of the object to the new image; the labeled new image comprising the training data; and

training a deep learning model using the training data.

16 . The computer system of claim 15 , wherein the labeling of the one or more objects comprises:
receiving an image; labeling an object in the image by recognizing the object using a trained deep learning model; identifying the set of points in the point cloud corresponding to the object; and labeling the set of points based on the label of the object.

17 . The computer system of claim 15 , wherein the set of points is identified using a bounding box.

18 . The computer system of claim 15 , wherein the labeling of the one or more objects further comprises generating training labels by propagating map features to labels.

19 . The computer system of claim 15 , wherein the labeling of the one or more objects further comprises selecting labels to review by applying filters to the labels and setting a sampling rate.

20 . The computer system of claim 15 , wherein the labeling of the one or more objects further comprises reviewing labels by verifying accuracy of labels.

21 . The computer system of claim 15 , wherein the labeling of the one or more objects further comprises creating a dataset by selecting a subset of labels for the training of the deep learning model.

Assignees

• Deepmap Inc

Inventors

• Zhang Yu
• Yang Lin

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G05D1/246

  using environment maps, e.g. simultaneous localisation and mapping [SLAM] · CPC title

• G01C21/3841 Primary

  Data obtained from two or more sources, e.g. probe vehicles · CPC title