Sensor-based camera motion detection for unconstrained slam

Patent metadata

Field	Value
Publication number	US-9390344-B2
Application number	US-201514645675-A
Country	US
Kind code	B2
Filing date	Mar 12, 2015
Priority date	Jan 9, 2014
Publication date	Jul 12, 2016
Grant date	Jul 12, 2016

Abstract

Techniques are presented for monocular visual simultaneous localization and mapping (SLAM) based on detecting a translational motion in the movement of the camera using at least one motion sensor, while the camera is performing panoramic SLAM, and initializing a three dimensional map for tracking of finite features. Motion sensors may include one or more sensors, including inertial (gyroscope, accelerometer), magnetic (compass), vision (camera) or any other sensors built into mobile devices.

First claim

What is claimed is:

1. The method for detecting translational motion while tracking a device in panoramic Simultaneous Localization and Mapping (SLAM) mode, comprising:
determining a vision-based rotational motion angle for the device, wherein the vision-based rotational motion angle is determined by performing image processing on a plurality of keyframes;
determining a sensor-based rotational motion angle for the device between the plurality of keyframes, wherein the sensor-based rotational motion angle is determined using one or more motion sensors coupled to the device;
determining a parallax angle for the device, wherein the parallax angle is determined by comparing the vision-based rotational motion angle and the sensor-based rotational motion angle;
comparing the parallax angle against a threshold value; and
detecting the translational motion based upon the parallax angle being above the threshold value.

2. The method of claim 1 , further comprising determining the translational motion using at least the parallax angle between the plurality of keyframes.

3. The method of claim 2 , further comprising determining depth information for one or more features with respect to the device by triangulating distance for the one or more features from the device using at least the translational motion.

4. The method of claim 1 , further comprising initializing a three dimensional (3D) map for tracking the device in a general motion mode, at least based on detecting of the translational motion.

5. The method of claim 1 , further comprising switching to six degrees of freedom (6DOF) SLAM mapping based on detecting the translational motion.

6. The method of claim 1 , further comprising initializing a three dimensional (3D) map for tracking the device using stereo initialization of the 3D map using two keyframes, at least based on detecting of the translational motion.

7. The method of claim 1 , wherein the vision-based rotational motion angle and the sensor-based rotational motion angle are detected for a motion between a reference keyframe acquired at a first time and a current keyframe acquired at a second time.

8. The method of claim 1 , wherein the one or more motion sensors include one or more of a gyroscope, accelerometer, or magnetometer.

9. A device for detecting translational motion while tracking a device in panoramic Simultaneous Localization and Mapping (SLAM) mode, the device comprises:
a memory;
a camera coupled to the device for obtaining information about the physical scene; and
a processor coupled to the memory and configured to:
determine a vision-based rotational motion angle for the device, wherein the vision-based rotational motion angle is determined by performing image processing on a plurality of keyframes;
determine a sensor-based rotational motion angle for the device between the plurality of keyframes, wherein the sensor-based rotational motion angle is determined using one or more motion sensors coupled to the device;
determine a parallax angle for the device, wherein the parallax angle is determined by comparing the vision-based rotational motion angle and the sensor-based rotational motion angle;
compare the parallax angle against a threshold value; and
detect the translational motion based upon the parallax angle being above the threshold value.

10. The device of claim 9 , further comprising the processor configured to determine the translational motion using at least the parallax angle between the plurality of keyframes.

11. The device of claim 10 , further comprising the processor configured to determine depth information for one or more features with respect to the device by triangulating distance for one or more features from the device using at least the translational motion.

12. The device of claim 9 , further comprising the processor initializing a three dimensional (3D) map for tracking the device in a general motion mode, at least based on detecting of the translational motion.

13. The device of claim 9 , further comprising the processor switching to six degrees of freedom (6DOF) SLAM mapping.

14. The device of claim 9 , further comprising the processor initializing a three dimensional (3D) map for tracking the device using stereo initialization of the 3D map using two keyframes, at least based on detecting of the translational motion.

15. The device of claim 9 , further comprising the processor initializing a three dimensional (3D) map for tracking the device by using a known tracking target, at least based on detecting of the translational motion.

16. The device of claim 9 , wherein the vision-based rotational motion angle and the sensor-based rotational motion angle are detected for a motion between a reference keyframe acquired at a first time and a current keyframe acquired at a second time.

17. The device of claim 9 , wherein the one or more motion sensors include one or more of a gyroscope, accelerometer, or magnetometer.

18. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises instructions executable by a processor for performing simultaneous localization and mapping (SLAM), the instructions comprising instructions to:
determine a vision-based rotational motion angle for a device, wherein the vision-based rotational motion angle is determined by performing image processing on a plurality of keyframes;
determine a sensor-based rotational motion angle for the device between the plurality of keyframes, wherein the sensor-based rotational motion angle is determined using one or more motion sensors coupled to the device;
determine a parallax angle for the device, wherein the parallax angle is determined by comparing the vision-based rotational motion angle and the sensor-based rotational motion angle;
compare the parallax angle against a threshold value; and
detect a translational motion based upon the parallax angle being above the threshold value.

19. The non-transitory computer-readable storage medium of claim 18 , further comprising determining the translational motion using at least the parallax angle between the plurality of keyframes.

20. The non-transitory computer-readable storage medium of claim 19 , further comprising the processor configured to determine depth information for one or more features with respect to the device by triangulating distance for the one or more features from the device using at least the translational motion.

21. The non-transitory computer-readable storage medium of claim 18 , wherein the vision-based rotational motion angle and the sensor-based rotational motion angle are detected for a motion between a reference keyframe acquired at a first time and a current keyframe acquired at a second time.

22. The non-transitory computer-readable storage medium of claim 18 , wherein the one or more motion sensors include one or more of a gyroscope, accelerometer, or magnetometer.

23. An apparatus, comprising:
means for determining a vision-based rotational motion angle for a device, wherein the vision-based rotational motion angle is determined by performing image processing on a plurality of keyframes;
means for determining a sensor-based rotational motion angle for the device between the plurality of keyframes, wherein the sensor-based rotational motion angle is determined using one or more motion sensors coupled to the device;
means for determining a parallax angle for the device, wherein the parallax angle is d

Assignees

• Qualcomm Inc

Inventors

• Pirchheim Christian
• Arth Clemens
• Schmalstieg Dieter

Classifications

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/004

  Physics · mapped topic

• G06T3/00

  Geometric image transformations in the plane of the image · CPC title

• G06T2207/30244

  Camera pose · CPC title