Constrained key frame localization and mapping for vision-aided inertial navigation

Patent metadata

Field	Value
Publication number	US-9996941-B2
Application number	US-201715470595-A
Country	US
Kind code	B2
Filing date	Mar 27, 2017
Priority date	May 8, 2013
Publication date	Jun 12, 2018
Grant date	Jun 12, 2018

Abstract

Estimation techniques for vision-aided inertial navigation are described. In one example, a vision-aided inertial navigation system (VINS) comprises an image source to produce image data for a keyframe and one or more non-keyframes along a trajectory, the one or more non-keyframes preceding the keyframe along the trajectory. The VINS comprises an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes, and a processing unit comprising an estimator that processes the IMU data and the image data to compute state estimates of the VINS. The estimator computes the state estimates of the VINS for the keyframe by constraining the state estimates based on the IMU data and the image data for the one or more non-keyframes of the VINS without computing state estimates of the VINS for the one or more non-keyframes.

First claim

What is claimed is:

1. A device comprising:
an image source configured to produce image data for a first keyframe, one or more nonkeyframes, and a second keyframe along a trajectory of the device, the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;
an inertial measurement unit (IMU) configured to produce IMU data indicative of a motion of the device along the trajectory; and
a processor configured to:
process the IMU data and the image data to compute respective state estimates for a position and orientation of the device for the first keyframe and for the second keyframe;
process the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe, wherein when computing the one or more constraints the processor is configured to treat a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes; and
apply the one or more constraints when computing the state estimates of the device for the second keyframe.

2. The device of claim 1 , wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe, the processor is configured to process both the IMU data and the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe.

3. The device of claim 1 , wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe, the processor is configured to:
process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe; and
discard IMU data associated with the non-keyframes.

4. The device of claim 1 , wherein the processor is further configured to compute each of the constraints as an estimate of motion from the first keyframe to the second keyframe, and a covariance indicating an uncertainty of the estimated motion.

5. The device of claim 1 , wherein the processor is further configured to compute the respective state estimates for the first keyframe and the second keyframe without computing state estimates for the position and the orientation of the device for each of the non-keyframes.

6. The device of claim 1 ,
wherein the processor is further configured to compute the state estimates to include a respective estimated position of each landmark observed for the first keyframe and the second keyframe, and
wherein, to compute the state estimates for the second keyframe, the processor is further configured to constrain the position and orientation of the device within the second keyframe based on the image data from the one or more non-keyframes while disregarding dependencies for each of the landmarks with respect to landmarks observed within the image data for the one or more non-keyframes.

7. The device of claim 1 , wherein the processor is configured to compute estimates for positions of landmarks observed from the first keyframe and the second keyframe without computing estimates for positions of landmarks observed only from the non-keyframes for which a respective pose of the device is not computed.

8. The device of claim 1 ,
wherein the processor is configured to maintain a state vector that specifies, for computation, state estimates of the position and the orientation of the device for the first keyframe and one or more landmarks observed from the first keyframe or the second keyframe without including variables for non-keyframe poses or landmarks observed only from non-keyframes,
wherein the processor is configured to iteratively process the state vector to compute the state estimates, and
wherein, for each iteration, the processor is configured to constrain updates for the state estimates for the position and orientation of the device and the landmarks observed from the keyframes for the device based on the image data for the one or more non-keyframes of the device.

9. The device of claim 8 , wherein the processor is further configured to:
select key frames along the trajectory for which respective state estimates are to be computed within the state vector, and
select the key frames based on a set of criteria comprising one or more of a distance traveled between two consecutive key poses and poses at which points of interest were detected within the image data.

10. The device of claim 1 , wherein the processor is configured to build a map of the environment to include the state estimates of the device.

11. The device of claim 1 , wherein the device is integrated within a mobile phone or a robot.

12. The device of claim 1 , wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device, the processor is further configured to process the image data associated with the non-keyframes to compute the one or more constraints without processing the IMU data to compute the one or more constraints.

13. The device of claim 1 , wherein the processor is further configured to compute, based on the computed state estimates of the device, a speed of the device.

14. The device of claim 1 , wherein the processor is further configured to compute, based on the computed state estimates of the device, an odometry of the device.

15. A system comprising:
an image source to produce image data for a first keyframe, one or more non-keyframes, and a second keyframe along a trajectory of a vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;
an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory; and
a processing unit comprising an estimator that processes the IMU data and the image data to compute respective state estimates for a position and orientation of the VINS for the first keyframe and for the second keyframe,
wherein the estimator processes the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe,
wherein, when computing the one or more constraints, the processor is configured to treat a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes, and
wherein the estimator applies the one or more constraints when computing the state estimates of the VINS for the second keyframe.

16. The system of claim 15 , wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second key

Assignees

• Univ Minnesota

Inventors

• Roumeliotis Stergios I
• Nerurkar Esha D

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30241

  Trajectory · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G01C21/165

  combined with non-inertial navigation instruments · CPC title