Method and apparatus for generating three-dimensional pose using multi-modal sensor fusion

Patent metadata

Field	Value
Publication number	US-9031809-B1
Application number	US-201113182987-A
Country	US
Kind code	B1
Filing date	Jul 14, 2011
Priority date	Jul 14, 2010
Publication date	May 12, 2015
Grant date	May 12, 2015

Abstract

A method and apparatus for providing three-dimensional navigation for a node comprising an inertial measurement unit for providing gyroscope, acceleration and velocity information (collectively IMU information); a ranging unit for providing distance information relative to at least one reference node; at least one visual sensor for providing images of an environment surrounding the node; a preprocessor, coupled to the inertial measurement unit, the ranging unit and the plurality of visual sensors, for generating error states for the IMU information, the distance information and the images; and an error-state predictive filter, coupled to the preprocessor, for processing the error states to produce a three-dimensional pose of the node.

First claim

The invention claimed is:

1. Apparatus for providing three-dimensional navigation for a node comprising:
an inertial measurement unit (IMU), coupled to the node, for providing gyroscope, acceleration and velocity information, collectively IMU information;
a ranging unit, coupled to the node, for providing distance information relative to at least one reference node;
at least one visual sensor, coupled to the node, for providing images of an environment surrounding the node wherein features are tracked in the images in adjacent frames and combined with the IMU information to produce relative pose measurements;
a landmark matcher for generating an error state derived from comparing at least one image to a visual landmark within the environment;
a preprocessor, coupled to the inertial measurement unit, the ranging unit, the at least one visual sensor and the landmark matcher for fusing the relative pose measurements, the error state from the landmark matcher, the distance information and the IMU information to generate an error state comprising local and global information; and
an error-state predictive filter, coupled to the preprocessor, for processing the error state, which comprises local and global information to produce a continually updated three-dimensional pose of the node,
wherein the error-state predictive filter localizes the node by estimating frame to frame information, estimating ranging information and estimating IMU information and integrating the estimates over time when the node fails to localize via the landmark matcher.

2. The apparatus of claim 1 wherein the at least one reference node is at least one of an anchor node or mobile node.

3. The apparatus of claim 1 wherein the at least one visual sensor comprises at least one stereo camera.

4. The apparatus of claim 1 wherein the visual landmark is recalled from a landmark database.

5. The apparatus of claim 4 wherein the landmark database is created by scanning the environment with a sensor to create a point cloud representation of the environment.

6. The apparatus of claim 1 further comprising an output processor for generating, in response to the pose, at least one of tracking information or an augmented reality display.

7. The apparatus of claim 1 wherein the error state further comprises range bias, camera location, gyroscope bias vector, velocity vector, accelerometer bias vector, and ground to camera orientation.

8. The apparatus of claim 7 wherein the error-state predictive filter produces the three dimensional pose when image information is inaccurate or not available.

9. The apparatus of claim 1 wherein the apparatus is attached to a helmet.

10. The apparatus of claim 1 wherein the error states comprise camera location, gyroscope bias vector, velocity vector, accelerometer bias vector, and ground to camera orientation.

11. The apparatus of claim 1 wherein the error- state predictive filter produces the three dimensional pose when image information is inaccurate or not available.

12. A method, executed by a processer, for generating a pose of a node, the method comprising:
using an inertial measurement unit coupled to the node for providing gyroscope, acceleration and velocity information, collectively IMU information, using a ranging unit coupled to the node for providing distance information relative to at least one reference node, and using at least one visual sensor coupled to the node for providing images of an environment surrounding the node, wherein features are tracked in adjacent frames in the images and combined with the IMU information to produce relative pose measurements;
generating, by a landmark matcher, an error state derived from comparing at least one image to a visual landmark within the environment;
fusing the relative pose measurements, the error state from the landmark matcher, the distance information and the IMU information to generate an error state comprising local and global information; and
processing, by the processor, the error state, which comprises local and global information, within an error state predictive filter to produce a continually three-dimensional pose of the node,
wherein the error state predictive filter localizes the node by estimating frame to frame information, estimating ranging information and estimating IMU information and integrating the estimates over time when the node fails to localize via the landmark matcher.

13. The method of claim 12 wherein the error state further comprises camera location, gyroscope bias vector, velocity vector, accelerometer bias vector, and ground to camera orientation.

14. The method of claim 12 wherein the error state predictive filter produces the three dimensional pose when image information is inaccurate or not available.

Assignees

• Kumar Rakesh
• Samarasekera Supun
• Chiu Han-Pang
• Zhu Zhiwei
• Oskiper Taragay
• Wang Lu
• Hadsell Raia
• Stanford Res Inst Int

Inventors

• Kumar Rakesh
• Samarasekera Supun
• Chiu Han-Pang
• Zhu Zhiwei
• Oskiper Taragay
• Wang Lu
• Hadsell Raia

Classifications

• G01C21/188 Primary

  for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems · CPC title

• G01C21/1656

  with passive imaging devices, e.g. cameras · CPC title

• G01C21/1652

  with ranging devices, e.g. LIDAR or RADAR · CPC title

• G06T7/0042

  Physics · mapped topic

• G01S1/00

  Beacons or beacon systems transmitting signals having a characteristic or characteristics capable of being detected by non-directional receivers and defining directions, positions, or position lines fixed relatively to the beacon transmitters; Receivers co-operating therewith (position fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00) · CPC title