3-D motion estimation and online temporal calibration for camera-IMU systems

Patent metadata

Field	Value
Publication number	US-10306206-B2
Application number	US-201414339276-A
Country	US
Kind code	B2
Filing date	Jul 23, 2014
Priority date	Jul 23, 2013
Publication date	May 28, 2019
Grant date	May 28, 2019

Abstract

A system and method for a 3-D motion estimation and online temporal calibration having one or more than one processor, one or more than one camera, one or more than one inertial sensor, a storage for storing data and instructions. The stored instructions comprise a filter estimation module, high-precision navigation instructions and consistent state estimates in scenarios involving features, with both constant and time-varying offsets from the filter estimation module.

First claim

What is claimed is:

1. A system for 3-D motion estimation of a moving platform including one or more than one camera and one or more than one inertial measurement unit, the system comprising:
one or more than one processor;
the one or more than one camera operably connected to the one or more than one processor and configured to produce at least one visual image;
the one or more than one inertial measurement unit operably connected to the one or more than one processor and configured to produce at least one inertial measurement;
a storage for storing data and instructions executable on the one or more than one processor comprising an estimation module that calculates online:
estimates of a time offset between time stamps associated with images from the one or more cameras and time stamps associated with the one or more inertial measurement units, wherein each of said time offset estimates is based on both of (i) one of the time stamps associated with images from the one or more cameras and (ii) one of the time stamps associated with the one or more inertial measurement units; and
employs the time offset estimates to estimate a state of the moving platform;

wherein the processor is configured to:
receive the at least one visual image from the one or more than one camera, the at least one inertial measurement from the one or more than one inertial measurement unit, at least one visual image timestamp for the at least one visual image, and at least one inertial measurement timestamp for the at least one inertial measurement;
calculate online the time offset from the at least one visual image timestamp and the at least one inertial measurement timestamp using the estimation module, wherein the estimation module generates Jacobian matrices with respect to the time offset; and
generate estimates of a state of the moving platform based on the calculated time offset.

2. The system of claim 1 , wherein the camera and the inertial measurement unit are in an environment containing features with known 3D coordinates.

3. The system of claim 1 , wherein the camera and the inertial measurement unit are in an environment containing features with a priori unknown 3D coordinates.

4. The system of claim 1 , wherein the time offset for each camera is the amount of time that the one or more than one camera timestamp is shifted, so that the one or more than one camera and the inertial measurement unit data streams become temporally consistent.

5. The system of claim 1 , where the system has a state and the state of the system comprises one or more than one of the position, orientation, velocity, of the system.

6. The system of claim 1 , where the system has a state and the state of the system includes a description of the spatial configuration between the one or more than one camera and the one or more than one inertial measurement unit.

7. A method for a 3-D motion estimation of a moving platform including one or more than one camera and one or more than one inertial measurement unit, the method comprising the steps of:
providing the system of claim 1 ;
instructions executable on the one or more than one processor for the steps of:
receiving at least one visual image from the one or more than one camera;
receiving at least one inertial measurement from the one or more than one inertial measurement unit;
receiving at least one visual image timestamp for the at least one visual image;
receiving at least one inertial measurement timestamp for the at least one inertial measurement;
calculating online a time offset estimate from both the at least one visual image timestamp and the at least one inertial measurement timestamp using an estimator; and
employing the calculated time offset estimate to estimate a state of the moving platform.

8. A method for a 3-D motion estimation of a moving platform including one or more than one camera and one or more than one inertial measurement unit, the method comprising the steps of:
a) providing a system for 3-D motion estimation of a moving platform, wherein the system comprises a processor comprising an estimation module, a camera, an inertial measurement unit, and a storage for storing data and instructions executable on the estimation module;
b) receiving at least one visual image from the camera;
c) receiving at least one inertial measurement from the inertial measurement unit;
d) receiving at least one visual image timestamp for the at least one visual image;
e) receiving at least one inertial measurement timestamp for the at least one inertial measurement;
f) calculating online a time offset from the at least one visual image timestamp and the at least one inertial measurement timestamp using a filter, wherein the filter is based on Jacobian matrices of the inertial measurements with respect to the time offset; and
g)generating estimates of a state of the moving platform based on the calculated time offset;
h) propagating the state estimates from inertial measurement unit measurements;
i) processing the least one visual image for updating the state estimates; and
j) modeling effects of errors in the time offset.

9. The method of claim 7 , wherein calculating a time-offset estimate is performed by an extended Kalman filter for simultaneous localization and mapping.

10. The method of claim 7 , wherein the system further comprises a filter, and the filter estimates:
an inertial measurement unit state, comprising an inertial measurement unit position, velocity, orientation, and biases;
the spatial transformation between the camera and inertial measurement unit frames;
time offsets between measurement unit data streams; and
positions of visual features.

11. The method of claim 8 , wherein the step of calculating explicitly includes the time offset in an extended-Kalman filter state vector.

12. The method of claim 8 , wherein step f) is performed without known feature positions.

13. The method of claim 8 further comprising the step of extracting motion information from feature measurements and fusing the extracted motion information with state estimates, inertial measurement unit measurements or both state estimates and inertial measurement unit measurements.

14. The method of claim 8 further comprising the step of concurrently calculating estimates of the time offset and the camera-to-inertial measurement unit extrinsics for visual-inertial odometry.

15. The method of claim 8 further comprising the step of calculating temporal and spatial alignment between the one or more than one camera and the one or more than one inertial measurement unit.

16. A method for a 3-D motion estimation of a moving platform including one or more than one camera and one or more than one inertial measurement unit, the method comprising the steps of:)
a) providing a system for 3-D motion estimation of a moving platform, wherein the system comprises a processor comprising an estimator, a camera, an inertial measurement unit, and a storage for storing data and instructions executable on the estimation module;
b) receiving at least one visual image from the camera;
c) receiving at least one inertial measurement from the inertial measurement unit;
d) receiving at least one visual image timestamp for the at least one visual image;
e) receiving at least one inertial measurement timestamp for the at least one inertial measurement;
f) calculating online a time offset from the at least one visual image timestamp and the at least one inertial measurement timestamp using the estimator, wherein the estimator generates Jacobian matrices with respect to the time offset;

Assignees

• Univ California

Inventors

• Mourikis Anastasios
• Li Mingyang

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G01C21/165

  combined with non-inertial navigation instruments · CPC title