Constrained key frame localization and mapping for vision-aided inertial navigation

Patent metadata

Field	Value
Publication number	US-9607401-B2
Application number	US-201414271971-A
Country	US
Kind code	B2
Filing date	May 7, 2014
Priority date	May 8, 2013
Publication date	Mar 28, 2017
Grant date	Mar 28, 2017

Abstract

Estimation techniques for vision-aided inertial navigation are described. In one example, a vision-aided inertial navigation system (VINS) comprises an image source to produce image data for a keyframe and one or more non-keyframes along a trajectory, the one or more non-keyframes preceding the keyframe along the trajectory. The VINS comprises an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes, and a processing unit comprising an estimator that processes the IMU data and the image data to compute state estimates of the VINS. The estimator computes the state estimates of the VINS for the keyframe by constraining the state estimates based on the IMU data and the image data for the one or more non-keyframes of the VINS without computing state estimates of the VINS for the one or more non-keyframes.

First claim

The invention claimed is:

1. A vision-aided inertial navigation system comprising:
an image source to produce image data for a first keyframe, one or more non-keyframes and a second keyframe along a trajectory of the vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;
an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes; and
a processing unit comprising an estimator that processes the IMU data and the image data to compute respective state estimates for a position and orientation of the VINS for the first keyframe and for the second keyframe,
wherein, when computing the state estimates, the estimator constrains the state estimates for the second keyframe relative to the state estimates for the first keyframe based on the IMU data and the image data from the one or more non-keyframes, and
wherein, when constraining the state estimates, the estimator treats a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes.

2. The vision-aided inertial navigation system of claim 1 ,
wherein the estimator processes the IMU data and the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe, and
where in the estimator applies the one or more constraints when computing the state estimates of the VINS for the second keyframe.

3. The vision-aided inertial navigation system of claim 2 , wherein the estimator computes each of the constraints as an estimate of motion from the first keyframe to the second keyframe, and a covariance indicating an uncertainty of the estimated motion.

4. The vision-aided inertial navigation system of claim 1 , wherein the estimator computes the respective state estimates for the first keyframe and the second keyframe without computing state estimates for the position and the orientation of the VINS for each of the non-keyframes.

5. The vision-aided inertial navigation system of claim 1 ,
wherein the estimator computes the state estimates to include a respective estimated position of each landmark observed for the first keyframe and the second keyframe, and
wherein, when computing the state estimates for the second keyframe, the estimator constrains the position and orientation of the VINS within the second keyframe based on the IMU data and the image data from the one or more non-keyframes while disregarding dependencies for each of the landmarks with respect to landmarks observed within the image data for the one or more non-keyframes.

6. The vision-aided inertial navigation system of claim 1 , wherein the processing unit computes estimates for positions of landmarks observed from the first keyframe and the second keyframe without computing estimates for positions of landmarks observed only from the non-keyframes for which a respective pose of the VINS is not computed.

7. The vision-aided inertial navigation system of claim 1 ,
wherein the processing unit maintains a state vector that specifies, for computation, state estimates the position and the orientation of the VINS for the first keyframe and one or more landmarks observed from the first keyframe or the second keyframe without including variables for non-keyframe poses or landmarks observed only from non-keyframes,
wherein the estimator unit iteratively processes the state vector to compute the state estimates, and
wherein, for each iteration, the estimator constrains updates for the state estimates for the position and orientation of the VINS and the landmarks observed from the keyframes for the VINS based on the IMU data and the image data for the one or more non-keyframes of the VINS.

8. The vision-aided inertial navigation system of claim 7 ,
wherein the estimator selects key frames along the trajectory for which respective state estimates are to be computed within the state vector, and
wherein the estimator selects the key frames based on a set of criteria comprising one or more of a distance traveled between two consecutive key poses and poses at which points of interest were detected within the image data.

9. The vision-aided inertial navigation system of claim 1 , wherein the estimator builds a map of the environment to include the state estimates of the VINS.

10. The vision-aided inertial navigation system of claim 1 , wherein the vision-aided inertial navigation system is integrated within a mobile phone or a robot.

11. A method for computing state estimates for a vision-aided inertial navigation system (VINS) comprising:
receiving image data produced by an image source of the vision-aided inertial navigation system for a first keyframe, one or more non-keyframes and a second keyframe along a trajectory of the vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;
receiving, from an inertial measurement unit (IMU), IMU data indicative of motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes; and
processing the IMU data and the image data to compute respective state estimates for a position and orientation of the VINS for the first keyframe and for the second keyframe,
wherein computing the state estimates comprises constraining the state estimates for the second keyframe relative to the state estimates for the first keyframe based on the IMU data and the image data from the one or more non-keyframes, and
wherein constraining the state estimates comprises treating a landmark observed within the image data for the first keyframe and the second keyframe as different from the same landmark observed within the image data for the non-keyframes.

12. The method of claim 11 , wherein computing the state estimates for the keyframe comprises:
processing the IMU data and the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe; and
applying the one or more constraints when computing the state estimates of the VINS for the keyframe.

13. The method of claim 12 , wherein processing the IMU data and the image data associated with the non-keyframes to compute one or more constraints comprises computing each of the constraints as (i) an estimate of motion from the first keyframe to the second keyframe, and (ii) a covariance indicating an uncertainty of the estimated motion.

14. The method of claim 11 , wherein computing the state estimates for the keyframe comprises computing the respective state estimates for the first keyframe and the second keyframe without computing state estimates for the position and the orientation of the VINS for each of the non-keyframes.

15. The method of claim 11 , wherein computing the state estimates for the keyframe comprises:
computing the state estimates to include a respective estimated position of each landmark observed for the first keyframe and the second keyframe, and
computing the state estimates for the second keyframe by constraining the position and orientation of the VINS within the second keyframe based on the IMU data and the image data from the one or more non-keyframes while disrega

Assignees

• Univ Minnesota

Inventors

• Roumeliotis Stergios I
• Nerurkar Esha D

Classifications

• G01C21/1656

  with passive imaging devices, e.g. cameras · CPC title

• G06T7/2033 Primary

  Physics · mapped topic

• G01C21/165

  combined with non-inertial navigation instruments · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30241

  Trajectory · CPC title