Method and apparatus for scale calibration and optimization of a monocular visual-inertial localization system

Patent metadata

Field	Value
Publication number	US-11756231-B2
Application number	US-202117362832-A
Country	US
Kind code	B2
Filing date	Jun 29, 2021
Priority date	Jun 29, 2021
Publication date	Sep 12, 2023
Grant date	Sep 12, 2023

Abstract

The method and system disclosed herein presents a method and system for capturing, by a camera disposed on a device moving in an environment, a plurality of image frames recorded in a first coordinate reference frame at respective locations within a portion of the environment in a first time period; capturing, by an inertial measurement unit disposed on the device, sets of inertial odometry data recorded in a second coordinate reference frame; determining a rotational transformation matrix that corresponds to a relative rotation between the first reference frame and the second reference frame; and determining a scale factor from the matching pairs of image frames. The rotational transformation matrix defines an orientation of the device, and the scale factor and the rotational transformation matrix calibrate the plurality of image frames captured by the camera.

First claim

What is claimed is:

1. A method, comprising:
capturing, by a camera disposed on a device moving in an environment, a plurality of image frames recorded in a first coordinate reference frame at respective locations within a portion of the environment in a first time period;
capturing, by an inertial measurement unit disposed on the device, sets of inertial odometry data recorded in a second coordinate reference frame, the sets of inertial odometry data corresponding to the plurality of image frames at the respective locations, in the first time period;
storing in a buffer, a matching pair of an image frame and a set of inertial odometry data that satisfies first criteria;
in accordance with a determination that a threshold number of matching pairs of image frames and inertial odometry data have been stored:
determining a rotational transformation matrix that corresponds to a relative rotation between the first coordinate reference frame and the second coordinate reference frame;
determining a scale factor from the matching pairs of image frames, wherein the rotational transformation matrix defines an orientation of the device, and the scale factor and the rotational transformation matrix calibrate the plurality of image frames captured by the camera, wherein the scale factor is solved as a parameter to a least square problem based on the matching pairs of image frames and inertial odometry data;

determining a translation transformation vector from the matching pairs of image frames, and wherein the rotational transformation matrix and the translation transformation vector define a transformation pose for transforming the first coordinate reference frame into the second coordinate reference frame, wherein the translation transformation vector is referenced to the first coordinate reference frame when a respective image of the matching pairs of image frames is taken to reduce scale ambiguity associated with the respective image,
multiplying the scale factor to the translation transformation vector, and
further multiplying a pose of a body frame of the device referenced to a camera frame when the respective image of the matching pairs of image frames is taken.

2. The method of claim 1 , wherein the translation transformation vector is solved as a parameter to a least square problem.

3. The method of claim 2 , further comprising:
determining a calibration quality for a respective image frame of the matching pairs of image frames by determining an error between an adjusted visual pose obtained through applying the scale factor, the rotation transformation matrix and the translation transformation vector to the respective image frame, and the set of inertial odometry data corresponding to the respective image frame.

4. The method of claim 3 , further comprising: capturing an additional sequence of image frames, and an additional sequence of sets to determine an updated scale factor until the calibration quality meets a second threshold.

5. The method of claim 1 , wherein determining the rotational transformation matrix comprises determining an inverse of a relative rotation between a body reference frame of the device and a camera reference frame of the camera.

6. The method of claim 1 , wherein the first criteria includes a relative distance between two adjacent sets of the inertial odometry data being larger than a minimum distance and being smaller than a maximum distance.

7. The method of claim 1 , further comprising determining an optimized scale factor by determining a residual error in the scale factor wherein the optimized scale factor includes multiplying the residual error with the scale factor.

8. The method of claim 7 , wherein determining the residual error comprises summing products of (i) relative translation between adjacent image frames and (ii) relative translation between adjacent sets of inertial odometry data.

9. The method of claim 7 , wherein the optimized scale factor is determined in accordance with a determination that the residual error in the scale factor is larger than a threshold value and that a number of optimized pairs of image frame and inertial odometry data set satisfies a second threshold.

10. The method of claim 9 , wherein a respective image frame in the optimized pairs of image frame is obtained by applying the scale factor and the rotational transformation matrix and a translation transformation vector on the respective image frame captured by the camera; and wherein an optimized pair of image frame and an inertial odometry data set is stored in a sliding window in accordance with a determination that the optimized pair meets a second criterion.

11. An electronic device, comprising:
one or more processing units;
memory; and
a plurality of programs stored in the memory that, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising:
capturing, by a camera disposed on a device moving in an environment, a plurality of image frames recorded in a first coordinate reference frame at respective locations within a portion of the environment in a first time period;
capturing, by an inertial measurement unit disposed on the device, sets of inertial odometry data recorded in a second coordinate reference frame, the sets of inertial odometry data corresponding to the plurality of image frames at the respective locations, in the first time period;
storing in a buffer, a matching pair of an image frame and a set of inertial odometry data that satisfies first criteria;
in accordance with a determination that a threshold number of matching pairs of image frames and inertial odometry data have been stored:
determining a rotational transformation matrix that corresponds to a relative rotation between the first coordinate reference frame and the second coordinate reference frame;
determining a scale factor from the number of matching pairs, wherein the rotational transformation matrix defines an orientation of the device, and the scale factor and the rotational transformation matrix calibrate the plurality of image frames captured by the camera, wherein the scale factor is solved as a parameter to a least square problem based on the matching pairs of image frames and inertial odometry data;

determining a translation transformation vector from the matching pairs of image frames, and wherein the rotational transformation matrix and the translation transformation vector define a transformation pose for transforming the first coordinate reference frame into the second coordinate reference frame, wherein the translation transformation vector is referenced to the first coordinate reference frame when a respective image of the matching pairs of image frames is taken to reduce scale ambiguity associated with the respective image,
multiplying the scale factor to the translation transformation vector, and
further multiplying a pose of a body frame of the device referenced to a camera frame when the respective image of the matching pairs of image frames is taken.

12. The electronic device of claim 11 , wherein the translation transformation vector is solved as a parameter to a least square problem.

13. The electronic device of claim 11 , wherein the plurality of programs causes the one or more processing units to perform operations further comprising determining an optimized scale factor by determining a residual error in the scale factor wherein the optimized scale factor includes multiplying the residual error with the scale factor.

14. The electronic device of claim 11 , wherein determining the rotational transformation matrix comprises determining

Assignees

• Midea Group Co Ltd

Inventors

• Chen Yi
• Huang Ke
• Xi Wei

Classifications

• G06T7/579 Primary

  from motion · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G05D1/0253

  extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70) · CPC title

• G05D1/027

  comprising intertial navigation means, e.g. azimuth detector (inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165) · CPC title

• G05D1/0272

  comprising means for registering the travel distance, e.g. revolutions of wheels (measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00) · CPC title