Object identification and labeling tool for training autonomous vehicle controllers

Patent metadata

Field	Value
Publication number	US-10175697-B1
Application number	US-201815906610-A
Country	US
Kind code	B1
Filing date	Feb 27, 2018
Priority date	Dec 21, 2017
Publication date	Jan 8, 2019
Grant date	Jan 8, 2019

Abstract

Techniques for identifying and labeling distinct objects within 3-D images of environments in which vehicles operate, to thereby generate training data used to train models that autonomously control and/or operate vehicles, are disclosed. A 3-D image may be presented from various perspective views (in some cases, dynamically), and/or may be presented with a corresponding 2-D environment image in a side-by-side and/or a layered manner, thereby allowing a user to more accurately identify groups/clusters of data points within the 3-D image that represent distinct objects. Automatic identification/delineation of various types of objects depicted within 3-D images, automatic labeling of identified/delineated objects, and automatic tracking of objects across various frames of a 3-D video are disclosed. A user may modify and/or refine any automatically generated information. Further, at least some of the techniques described herein are equally applicable to 2-D images.

First claim

What is claimed:

1. A computer-implemented method for identifying and labeling objects depicted within images for training machine-learning based models that are used to autonomously control vehicles, the method comprising:
displaying, on a user interface of one or more computing devices, an image of an environment in which vehicles operate, the image depicting one or more physical objects located in the environment;
providing, on the user interface, a lane-marking user control for use by a user to indicate lane markings within the image;
receiving, via the lane-marking user control, a user selection of a first location within the image that is non-adjacent to a second location within the image, the first location and the second location indicating respective endpoints of a segment of an edge of a traffic lane depicted within the image of the environment;
based upon the second location and the received user selection of the first location, automatically generating and displaying on the image, by the one or more computing devices, a marking indicative of the segment of the edge of the traffic lane; and
storing, by the one or more computing devices in one or more tangible, non-transitory memories, an indication of an association between a particular label for the traffic lane and data indicative of the segment of the edge of the traffic lane, thereby distinguishing the traffic lane from other areas and/or objects depicted within the image.

2. The computer-implemented method of claim 1 , wherein the segment of the edge of the traffic lane is a first segment of the edge of the traffic lane, the marking is a first marking, and the method further comprises:
automatically determining, by the one or more computing devices and based upon the data indicative of the first segment of the edge of the traffic lane, a second segment of the edge of the traffic lane, the second segment abutting the first segment, and the automatically determining the second segment including performing an image analysis of at least one of a first portion of the image abutting the first location or a second portion of the image abutting the second location;
automatically generating and displaying, by the one or more computing devices on the image, a second marking indicative of the second segment of the edge of the traffic lane; and
storing, in one or more memories, an indication of an association between the particular label for the traffic lane and data indicative of the second segment of the edge of the traffic lane, thereby further distinguishing the traffic lane from the other areas and/or objects depicted on the image.

3. The computer-implemented method of claim 2 , wherein:
the automatically generated and displayed second marking is a draft second marking;
the method further comprises receiving a user modification to the draft second marking; and
the data indicative of the second segment of the edge of the traffic lane is generated based on the modified, draft second marking.

4. The computer-implemented method of claim 1 , wherein:
receiving the user selection of the first location within the image comprises receiving the user selection of the first location within the image presented on the user interface from a first perspective view; and
the method further comprises:
adjusting, by the one or more computing devices in response to a user instruction, a presentation of the image displayed on the user interface to be from a second perspective view; and
automatically displaying, on the image presented on the user interface from the second perspective view, the marking indicative of the segment of the edge of the traffic lane that was displayed on the image from the first perspective view.

5. The computer-implemented method of claim 1 , further comprising receiving a user selection, via the lane-marking user control, of the second location within the image.

6. The computer-implemented method of claim 1 , wherein the image of the environment is a first frame depicting the environment at a first time, the segment of the edge of the traffic lane is a first segment of the edge of the traffic lane, the marking is a first marking, and the method further comprises:
presenting, on the user interface, a second frame depicting the environment at a second time different than the first time, the second frame displaying at least a portion of the first marking indicative of the first segment of the edge of the traffic lane;
determining, by the one or more computing devices and based on the data indicative of the first segment of the edge of the traffic lane, a second segment of the edge of the traffic lane as depicted in the second frame;
automatically generating and displaying, by the one or more computing devices, a second marking indicative of the second segment of the edge of the traffic lane; and
storing, in the one or more tangible, non-transitory memories, an indication of an association between the particular label for the traffic lane and data indicative of the second segment of the edge of the traffic lane, thereby distinguishing the traffic lane from other areas and/or objects depicted within the second frame.

7. The computer-implemented method of claim 6 , wherein determining the second segment of the edge of the traffic lane as depicted in the second frame comprises performing an image analysis of the environment as depicted within the second frame.

8. The computer-implemented method of claim 1 , wherein the image of the environment is a two-dimensional (2-D) visual image of the environment generated by one or more passive sensing devices, systems, or cameras.

9. The computer-implemented method of claim 1 , wherein the image of the environment comprises a dataset generated by one or more active sensing devices or systems.

10. The computer-implemented method of claim 9 , wherein the one or more active sensing devices or systems include one or more lidar devices and the dataset is a point cloud dataset.

11. The computer-implemented method of claim 1 , wherein the image of the environment is a three-dimensional (3-D) image of the environment.

12. The computer-implemented method of claim 1 , wherein storing the indication of the association between the particular label for the traffic lane and the data indicative of the label of the traffic lane comprises storing data indicative of the label in association with each pixel or data point included in the segment of the edge of the traffic lane depicted within the image.

13. The computer-implemented method of claim 1 , wherein the edge of the traffic lane is one of: a side or edge of a road, a lane delineator, or a lane divider.

14. The computer-implemented method of claim 1 , further comprising providing the stored indication of the association between the particular label for the traffic lane and the data indicative of the segment of the edge of the traffic lane as training data for a machine-learning model that executes to generate control signals to autonomously control one or more operations of a vehicle.

15. A system for identifying and labeling objects depicted within images for training machine-learning based models that are used to autonomously control vehicles, the system comprising:
a communication module;
one or more processors; and
one or more non-transitory, tangible memories coupled to the one or more processors and storing computer executable instructions thereon that, when executed by the one or more processors, cause the system to:
display, on a user interface, an image of an environment in which vehicles operate, the image depicting one or more physical objects located

Assignees

• Luminar Tech Inc

Inventors

• Sachdeva Prateek
• Trofymov Dmytro

Classifications

• G06V10/7788 Primary

  the supervisor being a human, e.g. interactive learning with a human teacher · CPC title

• G06T19/003

  Navigation within 3D models or images · CPC title

• G06T15/20

  Perspective computation · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N5/04

  Inference or reasoning models · CPC title