Autonomous navigation using visual odometry

Patent metadata

Field	Value
Publication number	US-9946264-B2
Application number	US-201615076745-A
Country	US
Kind code	B2
Filing date	Mar 22, 2016
Priority date	Mar 22, 2016
Publication date	Apr 17, 2018
Grant date	Apr 17, 2018

Abstract

A system and method are provided for autonomously navigating a vehicle. The method captures a sequence of image pairs using a stereo camera. A navigation application stores a vehicle pose (history of vehicle position). The application detects a plurality of matching feature points in a first matching image pair, and determines a plurality of corresponding object points in three-dimensional (3D) space from the first image pair. A plurality of feature points are tracked from the first image pair to a second image pair, and the plurality of corresponding object points in 3D space are determined from the second image pair. From this, a vehicle pose transformation is calculated using the object points from the first and second image pairs. The rotation angle and translation are determined from the vehicle pose transformation. If the rotation angle or translation exceed a minimum threshold, the stored vehicle pose is updated.

First claim

We claim:

1. A method for autonomously navigating a vehicle, the method comprising:
capturing a sequence of image pairs using a stereo camera;
providing a navigation application embedded in a non-transitory memory including a sequence of processor executable instructions for:
storing a vehicle pose, where a vehicle pose is defined as a position and orientation with respect to a coordination frame;
detecting a plurality of matching feature points in a first matching image pair;
determining a plurality of corresponding object points in three-dimensional (3D) space from the first image pair;
using a plane model, finding a dominant plane in the 3D space that fits the maximum number of points in the first image pair;
defining the dominant plane as an estimated ground plane in a world coordinate system;
in response to defining the estimated ground plane, determining vehicle camera height with respect to the estimated ground plane and camera pitch with respect to the estimated ground plane;
continuously redefining the estimated ground plane over a plurality of sequential image pairs as the vehicle changes position; wherein determining the camera height and camera pitch includes continuously redetermining the camera height and the camera pitch, such redetermining including avoiding the definition of a changing ground plane slope as an obstacle and finding the pitch angle sign as follows:
sign=(( N M )· V );

where N is a vector (0, 1, 0) normal to a plane in the camera coordinate system with y =0;
where M is a vector normal to the estimated ground plane in the camera coordinate system;
where V is a vector along the x axis (1, 0, 0) of the camera coordinate system;
where is a symbol representing a cross product operation;
tracking the plurality of feature points from the first image pair to a second image pair;
determining a plurality of corresponding object points in 3D space from the second image pair;
calculating a vehicle pose transformation using the object points from the first and second image pairs;
determining rotation angle and translation from the vehicle pose transformation; and,
when the rotation angle or translation exceed a minimum threshold, updating the stored vehicle pose.

2. The method of claim 1 wherein determining the rotation angle and translation from the vehicle pose transformation includes:
converting a rotation matrix to a rotation vector;
computing a length of the rotation vector to obtain a rotation angle; and,
computing a length of a 3×1 translation vector related to the captured image pairs to obtain vehicle movement.

3. The method of claim 1 wherein updating the stored vehicle pose when the rotation angle or translation exceed the minimum threshold includes minimizing the accumulation of errors in vehicle pose transformations calculated between image pairs.

4. The method of claim 3 wherein updating the stored vehicle pose when the rotation angle or translation exceed the minimum threshold includes discarding a vehicle transformation when the rotation angle and translation do not exceed the minimum threshold.

5. The method of claim 4 wherein updating the stored vehicle pose when the rotation angle or translation exceed the minimum threshold includes:
discarding a current vehicle pose transformation, and failing to update the stored vehicle pose, when the rotation angle and translation do not exceed the minimum threshold; and
updating the stored vehicle pose with a subsequent vehicle pose transformation in response to the rotation angle or translation exceeding the minimum threshold.

6. The method of claim 1 further comprising:
converting the vehicle pose into the world coordinate system having a ground plane.

7. The method of claim 1 wherein continuously redetermining the camera height and the camera pitch includes defining the estimated ground plane as invalid when the pitch angle exceeds a pitch angle maximum value.

8. The method of claim 1 further comprising:
removing all points on the estimated ground plane from the 3D space; and,
defining remaining points in the 3D space as potential obstacles.

9. The method of claim 1 wherein defining the estimated ground plane includes defining an estimated ground plane as points falling within a predetermined translation error and a predetermined rotation error of the points found in the dominant plane.

10. A system for autonomously navigating a vehicle, the system comprising:
a vehicle;
a stereo camera mounted on the vehicle having an output to supply image pairs, each image pair including a right image and a left image;
a processor;
a non-transitory memory comprising:
a stored vehicle pose, where a vehicle pose is defined as a position and orientation with respect to a coordination frame;
a navigation application embedded in the non-transitory memory including a sequence of processor executable instructions, the navigation application detecting a plurality of matching feature points in a first matching image pair, determining a plurality of corresponding object points in three-dimensional (3D) space from the first image pair, tracking the plurality of feature points from the first image pair to a second image pair, determining the plurality of corresponding object points in 3D space from the second image pair, calculating a vehicle pose transformation using the objects points from the first and second image pairs, determining rotation angle and translation from the vehicle pose transformation, and updating the stored vehicle pose when the rotation angle and translation exceed a minimum threshold;
wherein the navigation application:
converts the vehicle pose into a world coordinate system having a ground plane;
uses a plane model to find a dominant plane in the 3D space that fits the maximum number of points in an initial image frame, and defines the dominant plane as an estimated ground plane in the world coordinate system; and
finds a pitch angle sign of the camera as follows:
sign=(( N M )· V );

where N is a vector (0, 1, 0) normal to a plane in the camera coordinate system with y=0;
where M is a vector normal to the estimated ground plane in the camera coordinate system;
where V is a vector along the x axis (1, 0, 0) of the camera coordinate system; and,
where is a symbol representing a cross product operation.

11. The system of claim 10 wherein the navigation application determines the rotation angle and translation by converting a rotation matrix to a rotation vector, computing a length of the rotation vector to obtain a rotation angle, and computing a length of a 3×1 translation vector related to the captured image pairs to obtain vehicle movement.

12. The system of claim 10 wherein the navigation application minimizes the accumulation of errors in vehicle pose transformations calculated between image pairs.

13. The system of claim 12 wherein the navigation application discards a current vehicle pose transformation without updating the stored vehicle pose, when the rotation angle or translation do not exceed the minimum threshold, and updates the stored vehicle pose with a subsequent vehicle pose transformation in response to the rotation angle and translation exceeding the minimum threshold.

14. The system of claim 10 wherein the navigation application determines vehicle camera height with respect to the estimated ground plane, and camera pitch with respect to the estimated ground plane.

15. The system of claim 10 wherein the navigation application continuously redefines the estimated ground plane over a plurality of sequential image pairs as the vehicle

Assignees

• Sharp Laboratories America Inc

Inventors

• Liao Miao
• Li Ming
• Hong Soonhac

Classifications

• G06V10/757

  Matching configurations of points or features · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G05D2201/0213

  Physics · mapped topic

• G06K9/6211

  Physics · mapped topic

• G06K9/00805

  Physics · mapped topic