Object identification and labeling tool for training autonomous vehicle controllers

Patent metadata

Field	Value
Publication number	US-10535191-B2
Application number	US-201815906529-A
Country	US
Kind code	B2
Filing date	Feb 27, 2018
Priority date	Dec 21, 2017
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

Techniques for identifying and labeling distinct objects within 3-D images of environments in which vehicles operate, to thereby generate training data used to train models that autonomously control and/or operate vehicles, are disclosed. A 3-D image may be presented from various perspective views (in some cases, dynamically), and/or may be presented with a corresponding 2-D environment image in a side-by-side and/or a layered manner, thereby allowing a user to more accurately identify groups/clusters of data points within the 3-D image that represent distinct objects. Automatic identification/delineation of various types of objects depicted within 3-D images, automatic labeling of identified/delineated objects, and automatic tracking of objects across various frames of a 3-D video are disclosed. A user may modify and/or refine any automatically generated information. Further, at least some of the techniques described herein are equally applicable to 2-D images.

First claim

What is claimed:

1. A computer-implemented method for identifying and labeling objects within images for training machine-learning based models that are used to autonomously operate vehicles, the method comprising:
displaying, on a user interface of one or more computing devices, a three-dimensional (3-D) image of an environment in which vehicles operate, the 3-D image depicting one or more physical objects located in the environment;
providing, on the user interface, a paint user control for use by a user to indicate areas within images displayed on the user interface;
receiving, via a user activation of the paint user control, an indication of a data point within the 3-D image;
based upon the indicated data point within the 3-D image,
(i) automatically determining, without any additional user input aside from the user activation, boundaries of a depiction of an essentially planar surface area within the 3-D image, the depiction of the essentially planar surface area including the indicated data point and other data points surrounding the indicated data point within the 3-D image, and the essentially planar surface area being a particular type of surface area,
(ii) automatically determining, from a plurality of visual properties, each of which denotes a different type of surface area, a particular visual property corresponding to the particular type of the essentially planar surface area, and
(iii) automatically modifying, by the one or more computing devices and based on the determined boundaries, the particular visual property of the data points included within the boundaries of the depiction of the essentially planar surface area of the particular type;

obtaining, by the one or more computing devices, an indication of a particular label for the automatically determined depiction of the essentially planar surface area of the 3-D image; and
storing, by the one or more computing devices in one or more tangible, non-transitory memories, an indication of an association between data indicative of the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label, thereby distinguishing the essentially planar surface area from other physical objects depicted within the 3-D image.

2. The method of claim 1 , wherein the particular label is included in a plurality of labels of a plurality of areas and/or physical objects, and wherein each label included in the plurality of labels corresponds to a respective visual property.

3. The computer-implemented method of claim 1 , wherein the user activation of the paint user control consists of only a single mouse click.

4. The computer-implemented method of claim 1 , wherein the automatically determined depiction of the essentially planar surface area within the 3-D image is a first area representing a first essentially planar surface area of a first particular type, and the visual property is a first visual property, and the method further comprises modifying a second area within the 3-D image with a second visual property different than the first visual property, the second area being another subset of the total area of the 3-D image representing a second essentially planar surface area of a second particular type, and the modifying of the second area based on a usage of the paint user control as a virtual paint brush, a virtual paint spill, or another type of usage of the paint user control.

5. The computer-implemented method of claim 4 , wherein the usage of the paint user control is as the virtual paint brush, and modifying the second area with the second visual property comprises the user manipulating the virtual paint brush across the second area within the 3-D image.

6. The computer-implemented method of claim 5 , wherein a width of the virtual paintbrush is modifiable.

7. The computer-implemented method of claim 1 , wherein receiving, via the user activation of the paint user control, the indication of the data point within the 3-D image comprises receiving an indication of the user activation of the paint user control while a cursor or electronic pointer controlled by the paint user control hovers over or points to a location of the data point.

8. The computer-implemented method of claim 1 , further comprising receiving an indication of a particular type of object depicted in the 3-D image, the particular type of object corresponding to the essentially planar surface area.

9. The computer-implemented method of claim 1 , wherein the essentially planar surface area includes a road.

10. The computer-implemented method of claim 1 , wherein the essentially planar surface area includes a ground.

11. The computer-implemented method of claim 1 , wherein automatically modifying the particular visual property of the data points included within the depiction of the essentially planar surface area comprises automatically applying the particular visual property to the data points included within the depiction of the essentially planar surface area.

12. The computer-implemented method of claim 11 , further comprising receiving, via the paint user control, an indication of a portion of the data points included within the depiction of the essentially planar surface area from which the particular visual property is to be removed.

13. The computer-implemented method of claim 1 , wherein automatically modifying the particular visual property of the data points included within the depiction of the essentially planar surface area comprises removing the particular visual property from the data points included within the depiction of the essentially planar surface area of the 3-D image.

14. The computer-implemented method of claim 1 , wherein automatically determining the boundaries of the depiction of the essentially planar surface area within the 3-D image comprises automatically determining the boundaries of the depiction of the essentially planar surface area within the 3-D image by utilizing an object identification model that has been trained based on identified objects depicted within a plurality of historical images of one or more environments in which vehicles operate.

15. The computer-implemented method of claim 1 , wherein 3-D image of the environment comprises a dataset generated by one or more active sensing devices or systems.

16. The computer-implemented method of claim 15 , wherein the one or more active sensing devices or systems include one or more light detection and ranging (LIDAR) devices.

17. The computer-implemented method of claim 15 , wherein dataset is a point cloud dataset.

18. The computer-implemented method of claim 1 , wherein the 3-D environment image is an interactive, three-dimensional (3-D) image of the environment.

19. The computer-implemented method of claim 1 , wherein storing the indication of the association between the data indicative of the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label comprises storing data indicative of the particular label in association with each pixel or data point included in the automatically determined depiction of the essentially planar surface area of the 3-D image.

20. The computer-implemented method of claim 1 , wherein the method further comprises:
receiving, via the user interface, an instruction to hide respective images of one or more specific objects depicted within the 3-D image of the environment; and
based on the received hiding instruction, rendering non-visible the respective images of the one or m

Assignees

• Luminar Tech Inc

Inventors

• Sachdeva Prateek
• Trofymov Dmytro

Classifications

• G06V10/7788 Primary

  the supervisor being a human, e.g. interactive learning with a human teacher · CPC title

• G06T19/003 Primary

  Navigation within 3D models or images · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06T15/20

  Perspective computation · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title