Method and system for robust curb and bump detection from front or rear monocular cameras

Patent metadata

Field	Value
Publication number	US-10108866-B2
Application number	US-201615165070-A
Country	US
Kind code	B2
Filing date	May 26, 2016
Priority date	May 26, 2016
Publication date	Oct 23, 2018
Grant date	Oct 23, 2018

Abstract

A method of detecting a curb. An image of a path of travel is captured by a monocular image capture device mounted to a vehicle. A feature extraction technique is applied by a processor to the captured image. A classifier is applied to the extracted features to identify a candidate region in the image. Curb edges are localized by the processor in the candidate region of the image by extracting edge points. Candidate curbs are identified as a function of the extracted edge points. A pair of parallel curves is selected representing the candidate curb. A range from image capture device to the candidate curb is determined. A height of the candidate curb is determined. A vehicle application is enabled to assist a driver in maneuvering a vehicle utilizing the determined range and depth of the candidate curb.

First claim

What is claimed is:

1. A method of detecting a curb comprising:
capturing an image of a path of travel by an image capture device mounted to a vehicle, wherein an optical axis of the image capture device is parallel to a plane of a surface having the curb, and the image capture device has an XYZ camera-center reference system;
applying a feature extraction technique by a processor to the captured image to generate extracted features of the captured image;
applying a classifier to the extracted features to identify a candidate region in the image;
localizing curb edges by the processor in the candidate region of the captured image by extracting edge points;
identifying at least one candidate curb as a function of the extracted edge points;
selecting a pair of parallel curves representing the candidate curb;
determining a range from the image capture device to the candidate curb via the processor;
and
enabling a vehicle application to assist a driver in maneuvering a vehicle utilizing the determined range to the candidate curb;
wherein determining the range includes using the following formula:

D
g

=

h
⁢

w
.

g

u
.

g

where D g is the range, {dot over (w)} g and {dot over (u)} g are the coordinates along respective axes Z and X, in the XYZ camera-center reference system, of a bearing vector b=({dot over (u)},{dot over (v)},{dot over (w)}) of a point on the image, and h is a height of the image capture device above a road surface having the candidate curb.

2. The method of claim 1 wherein applying a feature extraction technique to the captured image includes applying a descriptor for dividing the captured image into cells.

3. The method of claim 2 wherein applying the descriptor includes applying a Histogram of Gradient for dividing the captured image into cells, and wherein a histogram of gradient directions is compiled for each cell as a function of the pixels in each cell.

4. The method of claim 3 wherein the cells have patches defined as small connected regions, and wherein applying a classifier to the extracted features includes applying a binary classifier to classify each of the patches of each cell to identify candidate regions.

5. The method of claim 3 wherein the cells have patches defined as small connected regions, and applying a classifier to the extracted features includes applying a support vector machine to classify each of the patches of each cell to identify candidate regions.

6. The method of claim 1 wherein localized curb edges in the candidate regions includes extracting edge points from the candidate regions.

7. The method of claim 6 wherein a Gaussian filter is applied to smooth the image prior to extracting the edge points.

8. The method of claim 1 wherein identifying candidate curb includes identifying a plurality of candidate curbs that are spaced within a predetermined range of one another.

9. The method of claim 8 wherein the plurality of candidate curbs include five curb lines.

10. The method of claim 8 wherein second order polynomials are used to fit curb lines in the candidate regions.

11. The method of claim 1 wherein the selected pair of parallel curves represent an upper boundary and a lower boundary of the candidate curb.

12. The method of claim 1 wherein determining a range from the image capture device to the candidate curb includes determining a range from a camera plane of the image capture device to the candidate curb.

13. The method of claim 1 further comprising applying temporal smoothing via the processor as a post-processing step to eliminate false positives.

14. The method of claim 13 wherein applying temporal smoothing includes capturing a next image at a next time frame, via the image capture device, and applying a tracking filter to the processed image and the next image via the processor, wherein the temporal smoothing observes redundancies between consecutive image frames.

15. The method of claim 1 wherein enabling a vehicle application includes displaying the image on a display device to the driver and highlighting the identified curb in the display device.

16. The method of claim 1 wherein enabling a vehicle application includes applying the range the candidate curb to an autonomous parking application, the autonomous parking application actuating vehicle devices to park the vehicle.

17. The method of claim 1 wherein enabling a vehicle application includes applying the range to the candidate curb to a collision avoidance application, the collision avoidance application actuating vehicle devices to avoid a collision with the curb.

18. The method of claim 1 wherein enabling a vehicle application includes applying the range to the candidate curb to a clear path detection system, the clear path detection system actuating vehicle devices to maintain the vehicle along the path of travel.

19. The method of claim 1 wherein enabling a vehicle application includes applying the range to the candidate curb to a lane centering application, the lane centering application actuating vehicle devices to center the vehicle within the lane.

20. A method of detecting a curb from a vehicle using an image capture device mounted to the vehicle and having an XYZ camera-center reference system, the method comprising:
capturing an image of a path of travel via the image capture device;
applying a feature extraction technique to the captured image using a processor to thereby generate extracted features of the image;
applying a classifier to the extracted features to identify a candidate region in the image;
localizing curb edges in the identified candidate region of the image by extracting edge points via the processor;
identifying a candidate curb in the identified candidate region as a function of the extracted edge points;
selecting a pair of parallel curves representing the candidate curb;
determining a range from the image capture device to the candidate curb via the processor; and
enabling a vehicle application to assist a driver in maneuvering a vehicle utilizing the determined range of the candidate curb;
wherein determining the range includes using the following formula:

D
g

=

f
⁢

f

u
.

g

where D g is the determined range, {dot over (u)} g is the coordinate along axis X within the XYZ camera-center reference system of a bearing vector b=({dot over (u)},{dot over (v)},{dot over (w)}) of a point on the image, h is a height of the camera above a road surface having the candidate curb, and f is the camera focal length provide

Assignees

• Gm Global Tech Llc
• Univ Carnegie Mellon
• Gm Global Tech Operations Llc

Inventors

• Prinet Veronique C
• Wang Jinsong
• Jongho Lee
• Wettergreen David

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/44 Primary

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06V10/50

  by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis · CPC title