Systems and methods for visual classification with region proposals

Patent metadata

Field	Value
Publication number	US-10460180-B2
Application number	US-201715492760-A
Country	US
Kind code	B2
Filing date	Apr 20, 2017
Priority date	Apr 20, 2017
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

Systems and method are provided for controlling an autonomous vehicle. A camera configured to capture an image, and a controller can execute an autonomous driving system (ADS) that classify that image. The ADS comprises a classification system for classifying objects in an environment within a driveable area of the autonomous vehicle. The classification system comprises a processor configured to execute a region proposal generator module and an image classification module. The region proposal generator module generates a set of bounding box region proposals for the image. The bounding box region proposals are selected areas of the image that include objects to be classified. The image classification module classifies, via a neural network executed by the processor, the objects from the image that are within one of the bounding box region proposals.

First claim

What is claimed is:

1. A method for classifying objects in an environment, the method comprising:
(a) processing, ranging information from depth sensors to identify a set of bounding box region proposals for an image from a camera, wherein the ranging information comprises point cloud data provided from lidar devices, wherein the bounding box region proposals are selected areas of the image that include objects to be classified, wherein each of the bounding box region proposals comprises: a set of two-dimensional bounding box coordinates that correspond to a region of a rectified image that includes one or more objects to be classified, wherein the set of bounding box region proposals collectively specify which objects are in the rectified image and where those objects are in the rectified image, wherein the processing comprises:
(a1) generating segmented objects based on the ranging information, wherein the segmented objects define three-dimensional locations and dimensions of objects in vicinity of a vehicle;
(a2) determining a subset of segmented objects to be classified that: meet a size constraint, are within a certain height range above the ground and are within a driveable area of the vehicle, and wherein other segmented objects that are not to be classified are those that the vehicle cannot hit and are disregarded when generating the object state information;
(a3) generating object state information that indicates the three-dimensional locations of the subset of segmented objects to be classified; and
(a4) translating the three-dimensional locations of objects as specified by the object state information into the set of bounding box region proposal; and
(b) classifying, via a neural network executed by the hardware based processor, only the objects from the image that are within one of the bounding box region proposals and generating an object classification result for each object from the rectified image that is within one of the bounding box region proposals;
(c) processing image data received from cameras to generate the rectified images; and
(d) generating a regressed bounding box for each object that is classified, wherein each regressed bounding box for each object is the bounding box that the neural network has determined to be a best bounding box encompassing that object, wherein steps (a1) through (a4), (b), (c) and (d) are performed iteratively such that the object classification result and the regressed bounding box for each object being classified are fed back on each iteration to refine the bounding box region proposal for each object during each subsequent iteration so that the bounding box region proposal for each object more closely describes actual geometry of that object.

2. The method according to claim 1 , wherein processing the object state information to generate the set of bounding box region proposals:
for each object:
projecting the three-dimensional location of that object relative to the vehicle into one of the rectified images.

3. The method according to claim 1 , wherein the ranging information further comprises at least one of:
radar data from radar devices;
stereo vision data from cameras that provides relative depth information; and
structured-light ranging data from a stereo vision system.

4. A classification system for classifying objects in an environment, the classification system comprising:
a hardware-based processor: and
memory comprising processor-executable instructions encoded on a non-transient processor-readable media, wherein the hardware-based processor is configurable to execute the processor-executable instructions to:
generate segmented objects based on ranging information, wherein the ranging information comprises point cloud data provided from lidar devices, wherein the segmented objects define three-dimensional locations and dimensions of objects in vicinity of a vehicle;
determine a subset of segmented objects to be classified that meet a size constraint based, are within a certain height range above the ground and are within a driveable area of the vehicle, wherein other segmented objects that are not to be classified are those that the vehicle cannot hit and are disregarded when generating the object state information; and generate object state information that indicates the three-dimensional locations of the subset of segmented objects with respect to the vehicle;
translate the three-dimensional locations of objects as specified by the object state information into a set of bounding box region proposals for an image from a camera, wherein the bounding box region proposals are selected areas of the image that include objects to be classified, wherein each of the bounding box region proposals comprises: a set of two-dimensional bounding box coordinates that correspond to a region of a rectified image that includes one or more objects to be classified, and wherein the set of bounding box region proposals collectively specify which objects are in the rectified image and where those objects are in the rectified image;
classify, via a neural network executed by the hardware based processor, only the objects from the rectified image that are within one of the bounding box region proposals and to disregard other objects from the rectified image such that only portions of the rectified image that are specified by one of the bounding box region proposals are analyzed to classify objects within the rectified image; and
generate an object classification result for each object from the rectified image that is within one of the bounding box region proposals; and
generate a regressed bounding box for each object that is classified, wherein each regressed bounding box for each object is the bounding box that the neural network has determined to be a best bounding box encompassing that object, wherein the object classification result and the regressed bounding box for each object being classified are fed back iteratively to refine the bounding box region proposal for each object during each subsequent iteration so that the bounding box region proposal for each object more closely describes actual geometry of that object.

5. An autonomous vehicle, comprising: a camera configured to capture an image; a controller comprising:
an autonomous driving system (ADS), comprising:
a classification system for classifying objects in an environment within a driveable area of the autonomous vehicle, the classification system comprising:
memory comprising processor-executable instructions encoded on a non-transient processor-readable media: and
a hardware-based processor to execute the processor-executable instructions to:
generate segmented objects based on ranging information, wherein the ranging information comprises point cloud data provided from lidar devices, wherein the segmented objects define three-dimensional locations and dimensions of objects in vicinity of the autonomous vehicle;
determine a subset of segmented objects to be classified that meet a size constraint based, are within a certain height range above the ground and are within a driveable area of the vehicle, wherein other segmented objects that are not to be classified are those that the vehicle cannot hit and are disregarded when generating the object state information;
generate object state information that indicates the three-dimensional locations of the subset of segmented objects with respect to the vehicle;
translate the three-dimensional locations of objects as specified by the object state information into a set of bounding box region proposals for the image, wherein the bounding box region proposals are selected areas of the image that include objects to be classified, wherein each of the bounding box region proposals comprises: a set of two-dimensional bounding box coordinates that

Assignees

• Gm Global Tech Operations Llc

Inventors

• Gao Peter
• Mou Wei
• Bhattacharya Siddhartho

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title