Object detection for an autonomous vehicle

Patent metadata

Field	Value
Publication number	US-10489686-B2
Application number	US-201715449501-A
Country	US
Kind code	B2
Filing date	Mar 3, 2017
Priority date	May 6, 2016
Publication date	Nov 26, 2019
Grant date	Nov 26, 2019

Abstract

An object detection system for an autonomous vehicle processes sensor data, including one or more images, obtained for a road segment on which the autonomous vehicle is being driven. The object detection system compares the images to three-dimensional (3D) environment data for the road segment to determine pixels in the images that correspond to objects not previously identified in the 3D environment data. The object detection system then analyzes the pixels to classify the objects not previously identified in the 3D environment data.

First claim

What is claimed is:

1. An object detection system for an autonomous vehicle (AV) comprising:
a memory to store an instruction set; and
one or more processors to execute instructions from the instruction set to:
process first sensor data obtained at a first time for a road segment on which the autonomous vehicle is being driven, wherein the processed first sensor data includes two or more images at two or more different angles captured at the first time;
compare the two or more images to create a disparity image;
process second sensor data obtained at a second time for the road segment on which the autonomous vehicle is being driven, wherein the processed second sensor data includes one or more images at the second time;
compare at least a first image of the two or more images at the first time and a first image of the one or more images at the second time to create a first optical flow image;
compare the disparity image and the first optical flow image to previously recorded three-dimensional (3D) environment data for the road segment to identify one or more areas that contain differences from the previously recorded 3D environment data for the road segment in a field of view of one or more sensors for the autonomous vehicle; and
analyze the identified one or more areas to classify objects in the field of view of the one or more sensors.

2. The object detection system of claim 1 , including further instructions that the one or more processors execute to:
identify a subset of sensor data from non-image sources corresponding to the one or more areas that contain differences from the previously recorded 3D environment data for the road segment; and
classify the objects based on analyzing the subset of sensor data.

3. The object detection system of claim 1 , including further instructions that the one or more processors execute to:
adjust operation of the autonomous vehicle based at least on a classification of the objects.

4. The object detection system of claim 1 , wherein the first optical flow image includes optical flow vectors calculated from the first image of the road segment at the first time and the second image of the road segment at the second time.

5. The object detection system of claim 1 , wherein the objects are classified into classes which include pedestrians, bicycles, and other vehicles.

6. The object detection system of claim 1 , including further instructions that the one or more processors execute to:
generate a baseline disparity image using the previously recorded 3D environment data for the road segment; and
compare the disparity image to the baseline disparity image to identify one or more areas of the disparity image that contains differences from the baseline disparity image.

7. The object detection system of claim 1 , wherein the processed second sensor data includes two or more images at the second time, the system including further instructions that the one or more processors execute to:
compare a second image of the two or more images at the first time with a second image of the two or more images at the second time to create a second optical flow image; and
compare the first optical flow image to the second optical flow image to classify the objects in the field of view of the one or more sensors.

8. The object detection system of claim 7 , including further instructions that the one or more processors execute to:
compare the second optical flow image to the previously recorded 3D environment data for the road segment to identify the one or more areas.

9. A method for object detection, the method being implemented by one or more processors of an autonomous vehicle and comprising:
processing first sensor data obtained at a first time for a road segment on which the autonomous vehicle is being driven, wherein the first processed sensor data includes two or more images at two or more different angles captured at the first time;
comparing the two or more images to create a disparity image;
processing second sensor data obtained at a second time for the road segment on which the autonomous vehicle is being driven, wherein the processed second sensor data includes one or more images at the second time;
comparing at least a first image of the two or more images at the first time and a first image of the one or more images at the second time to create a first optical flow image;
comparing the disparity image and the first optical flow image to previously recorded three-dimensional (3D) environment data for the road segment to identify one or more areas that contain differences from the previously recorded 3D environment data for the road segment in a field of view of one or more sensors for the autonomous vehicle; and
analyzing the identified one or more areas to classify objects in the field of view of the one or more images.

10. The method of claim 9 , further comprising:
identifying a subset of sensor data from non-image sources corresponding to the one or more areas that contain differences from the previously recorded 3D environment data for the road segment; and
classifying the objects based on analyzing the subset of sensor data.

11. The method of claim 9 , further comprising:
adjusting operation of the autonomous vehicle based at least on a classification of the objects.

12. The method of claim 9 , wherein the first optical flow image includes optical flow vectors calculated from the first image of the road segment at the first time and the second image of the road segment at the second time.

13. The method of claim 9 , wherein the objects are classified into classes which include pedestrians, bicycles, and other vehicles.

14. The method of claim 9 , further comprising:
generating a baseline disparity image using the previously recorded 3D environment data for the road segment; and
comparing the disparity image to the baseline disparity image to identify one or more areas of the disparity image that contains differences from the baseline disparity image.

15. The method of claim 9 , wherein the processed second sensor data includes two or more images at the second time, the method further comprising:
comparing a second image of the two or more images at the first time with a second image of the two or more images at the second time to create a second optical flow image; and
comparing the first optical flow image to the second optical flow image to classify the objects in the field of view of the one or more sensors.

16. The method of claim 15 , further comprising:
comparing the second optical flow image to the previously recorded 3D environment data for the road segment to identify the one or more areas.

17. A vehicle comprising:
one or more sensors to obtain sensor data from an environment around the vehicle;
a memory to store an instruction set; and
one or more processors to execute instructions from the instruction set to:
process first sensor data obtained for a road segment at a first time on which the vehicle is being driven, wherein the processed first sensor data includes two or more images at two or more different angles captured at the first time;
compare the two or more images to create a disparity image;
process second sensor data obtained at a second time for the road segment on which the autonomous vehicle is being driven, wherein the processed second sensor data includes one or more images at the second time;
compare at least a first image of the two or more images at the first time and a first image of the one or more images at the second time to create a first optical flow image

Assignees

• Uatc Llc

Inventors

• Vallespi-Gonzalez Carlos

Classifications

• G06F18/24 Primary

  Classification techniques · CPC title

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• B60R11/04

  Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title