Controlling an autonomous vehicle based upon labels assigned to objects represented in sensor data

Patent metadata

Field	Value
Publication number	US-2020057442-A1
Application number	US-201816103548-A
Country	US
Kind code	A1
Filing date	Aug 14, 2018
Priority date	Aug 14, 2018
Publication date	Feb 20, 2020
Grant date	—

Abstract

An autonomous vehicle controlled based upon the output of a trained object classifier is described herein. The object classifier is trained using labeled training data generated by a pipeline configured to assign labels to unlabeled sensor data. The pipeline includes transmitting sensor signal data capturing an object to individual computing devices for indications of an object type, wherein a label is assigned to the object based on the indications and provided to a data store as labeled training data. A learning system receives the labeled training data and generates a trained object classifier (e.g., a neural network) that is deployed in an autonomous vehicle to control operation of a mechanical system based on an output thereof.

First claim

What is claimed is:

1 . An autonomous vehicle comprising:
a vehicle propulsion system; a sensor system that is configured to output sensor signals; a computing system in communication with the vehicle propulsion system and the sensor system, wherein the computing system comprises:
a processor; and
memory that stores instructions that, when executed by the processor, cause the processor to perform acts comprising:
assigning a label to a first sensor signal output by the sensor system, the label being indicative of a type of an object captured in the first sensor signal, wherein the label is assigned to the first sensor signal based upon output of an object classifier system provided with the sensor signal, wherein the object classifier system is trained based upon labeled training data, and further wherein the labeled training data is generated through acts comprising:
receiving a second sensor signal;
transmitting the second sensor signal to computing devices operated by users who are trained to identify objects of the type;
receiving, from the computing devices, indications that the second sensor signal captures a second object of the type; and
assigning a second label to the second sensor signal based upon the indications, wherein the second label indicates that the second sensor signal captures the second object of the type, and further wherein the training data comprises the second sensor signal with the second label assigned thereto; and

controlling the vehicle propulsion system based upon the label assigned to the first sensor signal.

2 . The autonomous vehicle of claim 1 , wherein the indications received from the computing devices include a same indication received from a first computing device and a second computing device.

3 . The autonomous vehicle of claim 1 , wherein the indications received from the computing devices include different indications received from a first computing device and a second computing device; and
wherein a same indication as one of the first computing device or the second computing device is received from a third computing device included in the computing devices.

4 . The autonomous vehicle of claim 1 , wherein the indications received from the computing devices include different indications from a first computing device, a second computing device, and a third computing device; and
wherein a new label classification is proposed by the users who are trained to identify objects of the type.

5 . The autonomous vehicle of claim 1 , wherein the second sensor signal includes data provided from a data collection vehicle.

6 . The autonomous vehicle of claim 1 , wherein the computing devices display a plurality of objects captured in the second sensor signal, the plurality of objects identified as objects to receive a label.

7 . The autonomous vehicle of claim 1 , wherein at least one of the label or the second label is a multi-layered label that increases in granularity at each additional layer.

8 . The autonomous vehicle of claim 7 , wherein each layer of the multi-layered label is defined by indications from a different set of users who are trained to identify objects of the type.

9 . The autonomous vehicle of claim 7 , wherein a first layer of the multi-layered label is defined by at least one of a moving object label or a static object label.

10 . The autonomous vehicle of claim 1 , wherein a learning system receives the training data to generate the object classifier system.

11 . A method performed by an autonomous vehicle, the method comprising:
generating, by a sensor system, a first sensor signal; providing the first sensor signal to an object classifier system that is trained based upon labeled training data, wherein the labeled training data is generated through acts comprising:
receiving a second sensor signal;
transmitting the second sensor signal to computing devices operated by users who are trained to identify objects of a type;
receiving, from the computing devices, indications that the second sensor signal captures an object of the type; and
assigning a label to the second sensor signal based upon the indications, wherein the label indicates that the second sensor signal captures the object of the type, and further wherein the training data comprises the second sensor signal with the label assigned thereto;
assigning, by the object classifier system, a second label to the first sensor signal, the second label being indicative of a type of an object captured in the first sensor signal; and controlling a vehicle propulsion system based upon the second label assigned to the first sensor signal.

12 . The method of claim 11 , wherein receiving indications from the computing devices further comprises receiving a same indication from a first computing device and a second computing device.

13 . The method of claim 11 , wherein receiving indications from the computing devices further comprises receiving different indications from a first computing device and a second computing device; and
receiving, from a third computing device, a same indication as one of the first computing device or the second computing device.

14 . The method of claim 11 , wherein receiving indications from the computing devices further comprises receiving different indications from a first computing device, a second computing device, and a third computing device; and
proposing a new label classification by the users who are trained to identify objects of the type.

15 . The method of claim 11 , further comprising providing the second sensor signal from a data collection vehicle.

16 . The method of claim 11 , further comprising displaying a plurality of objects captured in the second sensor signal and identifying objects for labeling.

17 . The method of claim 11 , further comprising adding at least one layer to the label to provide a multi-layered label, wherein each layer of the multi-layered label defines the object of the type with increasing granularity.

18 . The method of claim 17 , further comprising providing indications from a different set of users who are trained to identify objects of the type to define each layer of the multi-layered label.

19 . The method of claim 11 , further comprising receiving the labeled training data by a learning system and generating the trained object classifier system.

20 . An autonomous vehicle comprising:
a computer-readable storage medium that comprises instructions that, when executed by one of more graphics processing units (GPUs), cause the one or more GPUs to perform actions comprising: generating, by a sensor system, a first sensor signal; providing the first sensor signal to an object classifier system that is trained based upon labeled training data, wherein the labeled training data is generated through acts comprising:
receiving a second sensor signal;
transmitting the second sensor signal to computing devices operated by users who are trained to identify objects of a type;
receiving, from the computing devices, indications that the second sensor signal captures an object of the type; and
assigning a label to the second sensor signal based upon the indications, wherein the label indicates that the second sensor signal captures the object of the type, and further wherein the training data comprises the second sensor signal with the label assigned thereto;
assigning, by the object classifier system, a second label to the first sensor signal, the second label being indicative of a

Assignees

• Gm Global Tech Operations Llc

Inventors

• Deiters Matthew
• Johnson Quinn

Classifications

• B60W2556/25

  Data precision · CPC title

• B60W2556/45

  External transmission of data to or from the vehicle · CPC title

• B60W2556/35

  Data fusion · CPC title

• B60W60/001 Primary

  Planning or execution of driving tasks · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title