Controlling an autonomous vehicle based upon labels assigned to objects represented in sensor data

Patent metadata

Field	Value
Publication number	US-10754338-B2
Application number	US-201816103548-A
Country	US
Kind code	B2
Filing date	Aug 14, 2018
Priority date	Aug 14, 2018
Publication date	Aug 25, 2020
Grant date	Aug 25, 2020

Abstract

An autonomous vehicle controlled based upon the output of a trained object classifier is described herein. The object classifier is trained using labeled training data generated by a pipeline configured to assign labels to unlabeled sensor data. The pipeline includes transmitting sensor signal data capturing an object to individual computing devices for indications of an object type, wherein a label is assigned to the object based on the indications and provided to a data store as labeled training data. A learning system receives the labeled training data and generates a trained object classifier (e.g., a neural network) that is deployed in an autonomous vehicle to control operation of a mechanical system based on an output thereof.

First claim

What is claimed is:

1. An autonomous vehicle comprising:
a vehicle propulsion system;
a sensor system that is configured to output sensor signals;
a computing system in communication with the vehicle propulsion system and the sensor system, wherein the computing system comprises:
a processor; and
memory that stores instructions that, when executed by the processor, cause the processor to perform acts comprising:
assigning a label to a first sensor signal output by the sensor system, the label being indicative of a type of an object captured in the first sensor signal, wherein the label is assigned to the first sensor signal based upon output of an object classifier system provided with the sensor signal, wherein the object classifier system is trained based upon labeled training data, and further wherein the labeled training data is generated through acts comprising:
receiving, at a remote computing system, a second sensor signal output by a second sensor system coupled to a vehicle;
selecting, by the remote computing system and from amongst several client computing devices, a first client computing device operated by a first user and a second client computing device operated by a second user, wherein the first client computing device and the second client computing device are selected due to the first user and the second user being trained to identify objects of the type in sensor signals output by sensor systems;
transmitting, by the remote computing system, the second sensor signal to the first client computing device and the second client computing device upon the remote computing system selecting the first client computing device and the second client computing device;
receiving, at the remote computing system and from both the first client computing device and the second client computing device, indications that the second sensor signal captures a second object of the type; and
assigning, by the remote computing system, a second label to the second sensor signal based upon the indications, wherein the second label indicates that the second object captured in the second sensor signal is of the type, and further wherein the training data comprises the second sensor signal with the second label assigned thereto; and

controlling the vehicle propulsion system based upon the label assigned to the first sensor signal.

2. The autonomous vehicle of claim 1 , wherein the first computing device and the second computing device display the second signal on displays of the first computing device and the second computing device, and further wherein the second object is highlighted to indicate that the first user and the second user are to assign a label to the second object.

3. The autonomous vehicle of claim 1 , wherein the labeled training data is generated through additional acts comprising:
upon assigning, by the remote computing system, the second label to the second sensor signal, selecting, by the remote computing system and from amongst the several client computing devices, a third client computing device operated by a third user and a fourth client computing device operated by a fourth user, wherein the third client computing device and the fourth client computing device are selected by the remote computing system due to the third user and the fourth user being trained to identify objects of a sub-type of the type in sensor signals output by sensor systems;
transmitting, by the remote computing system, the second sensor signal to the third client computing device and the fourth client computing device upon the remote computing system selecting the third client computing device and the fourth client computing device;
receiving, at the remote computing system and from both the third computing device and the fourth client computing device, second indications that the second object is of the sub-type of the type; and
assigning, by the remote computing system, a third label to the second sensor signal based upon the second indications, wherein the third label indicates that the second object captured in the second sensor signal is of the sub-type of the type.

4. The autonomous vehicle of claim 3 , wherein the third label is a multi-layered label that identifies the type of the second object and the sub-type of the second object.

5. The autonomous vehicle of claim 4 , wherein the type of the object is one of a moving object or a static object.

6. The autonomous vehicle of claim 1 , wherein a learning system receives the training data to generate the object classifier system.

7. The autonomous vehicle of claim 1 , wherein the labeled training data is generated through acts further comprising:
prior to selecting the second client computing device, selecting, by the remote computing system, a third client computing device from amongst the several client computing devices, wherein the third client computing device is operated by a third user, and further wherein the third client computing device is selected due to the third user being trained to identify objects of the type in the sensor signals output by the sensor systems;
prior to transmitting, by the remote computing system, the second sensor signal to the second client computing device, transmitting, by the remote computing system, the second sensor signal to the third client computing device;
receiving, at the remote computing system and from the third computing client computing device, an indication that the second object in the second sensor signal is of a second type that is different from the type; and
only after receiving the indication from the third computing device, selecting the second client computing device from amongst the several client computing devices.

8. The autonomous vehicle of claim 1 , wherein the type of the second object is amongst a predefined hierarchy of object types.

9. A method for training a computer-implemented classifier that is configured to assign labels to objects in sensor signals output by sensor systems of an autonomous vehicle, wherein the method is performed by a computing system, the method comprising:
receiving an image generated by a camera of a vehicle, wherein the image is of surroundings of the vehicle as the vehicle navigated over a roadway, and further wherein the image captures an object;
selecting, from at least three client computing devices, a first client computing device operated by a first user and a second client computing device operated by a second user, wherein the first client computing device and the second client computing device are selected due to the first user and the second user being trained to identify objects of a predefined type in images output by cameras;
upon selecting the first client computing device and the second client computing device, transmitting the image to the first client computing device and the second client computing device;
subsequent to transmitting the image to the first client computing device and the second client computing device, receiving indications from both the first client computing device and the second client computing device that the object captured in the image is of the predefined type;
only after receiving the indications from both the first client computing device and the second client computing device, assigning a label to the image, the label noting that the object captured in the image is of the predefined type; and
training an object classifier system based upon the image and the label assigned to the image, wherein the object classifier system is trained to identify objects of the predefined type as an autonomous vehicle navigates roadways, and further wherein a vehicle propulsion system of the autonomous vehicle is controlled based upon output of the object

Assignees

• Gm Global Tech Operations Llc

Inventors

• Deiters Matthew
• Johnson Quinn

Classifications

• B60W60/001 Primary

  Planning or execution of driving tasks · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G05D1/0088 Primary

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• B60W2556/45

  External transmission of data to or from the vehicle · CPC title