Method for estimating free space using a camera system

Patent metadata

Field	Value
Publication number	US-9305219-B2
Application number	US-201414162370-A
Country	US
Kind code	B2
Filing date	Jan 23, 2014
Priority date	Jan 23, 2014
Publication date	Apr 5, 2016
Grant date	Apr 5, 2016

Abstract

A method estimates free space near a moving object from a sequence of images in a video acquired of a scene by a camera system arranged on the moving object by first constructing a one-dimensional graph, wherein each node corresponds to a column of pixels in the image. Features are determined in the image, and an energy function is constructed on the graph based on the features. Using dynamic programming, the energy function is maximized to obtain the free space.

First claim

We claim:

1. A method for estimating free space near a moving object, comprising:
acquiring a sequence of images of a scene by a monocular camera system arranged on the moving object, and for each image in the sequence of images:
constructing a Markov random field as a one-dimensional graph, wherein each node in the graph corresponds to a discrete variable for a column of pixels in the image;
determining features in the image;
constructing an energy function on the one-dimensional graph based on the determined features; and
using dynamic programming to maximize the energy function to obtain a curve, wherein an area under the curve defines the free space near the object, wherein the free space is used for autonomous navigation of the moving object moving from one location to another, and wherein the steps are performed in a processor connected to the monocular camera system,
wherein the energy function comprises a sum of unary and pairwise potential functions, and wherein each potential function is determined using a corresponding feature among the determined features and a corresponding weight parameter learned from a sequence of training images.

2. The method of claim 1 , wherein the moving object is a vehicle and the scene includes a road.

3. The method of claim 1 , wherein the moving object is a boat and the scene includes water.

4. The method of claim 1 , further comprising:
estimating, for each column of pixels in the image, vertical coordinates of an obstacle.

5. The method of claim 1 , wherein the free space is in front of or behind the moving object.

6. The method of claim 4 , further comprising:
using an estimate of a horizon to restrict the vertical coordinates of the obstacle.

7. The method of claim 6 , wherein the estimate of the horizon is determined based on a configuration of the camera system.

8. The method of claim 6 , wherein the estimate of the horizon is determined using a set of training images.

9. The method of claim 1 , wherein the weight parameter is set manually.

10. The method of claim 1 , wherein the weight parameter is determined automatically from a set of training images using a structured support vector machine.

11. The method of claim 1 , wherein the features are selected from a group consisting of edges, appearance, homography, geometric context, smoothness, optical flow, a depth map and combinations thereof.

12. The method of claim 11 , wherein the appearance is modeled by Gaussian mixture models.

13. The method of claim 1 , further comprising:
determining a location prior probability for each pixel in the image to smooth the curve.

14. The method of claim 11 , wherein the homography imposes smoothness across the images, wherien the homography is based on a ground plane in the image.

15. The method of claim 1 , wherein a truncated quadratic penalty is used to smooth the curve.

16. The method of claim 1 , wherein the moving object is a vehicle, and wherein the free space is used for autonomous navigation of the vehicle moving from one location to another.

17. The method of claim 1 , wherein the moving object is a boat, and wherein the free space is used for manouvering the boat during berthing or mooring .

18. The method of claim 1 , wherein the moving object is a vehicle, and wherein the free space is used for parking assistance for the vehicle.

19. The method of claim 1 , wherein the moving object is a an indoor mobile robot, and wherein the free space is used by the indoor mobile robot for moving inside a building.

20. The method of claim 1 , wherein the moving object is a vacuum cleaning robot, and wherein the free space is used by the vacuum cleaning robot.

21. A system for estimating free space near a moving object comprising:
a monocular camera system arranged on the moving objects for acquiring a sequence of images of a scene; and
a processor connected to the monocular camera system being operable to:
construct a Markov random field as a one-dimensional graph, wherein each node in the graph corresponds to a discrete variable for a column of pixels in the image,
determine features in the image,
construct an energy function on the one-dimensional graph based on the determined features; and
use dynamic programming to maximize the energy function to obtain a curve, wherein an area under the curve defines the free space near the object, wherein the free space is used for autonomous navigation of the moving object moving from one location to another,

wherein the energy function comprises a sum of unary and pairwise potential functions, and wherein each potential function is determined using a corresponding feature among the determined features and a corresponding weight parameter learned from a sequence of training images.

Assignees

• Mitsubishi Electric Res Lab

Inventors

• Ramalingam Srikumar
• Yao Jian
• Taguchi Yuichi

Classifications

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06V10/85

  Markov-related models; Markov random fields · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06T11/26

  Drawing of charts or graphs · CPC title