System and method for hybrid simultaneous localization and mapping of 2D and 3D data acquired by sensors from a 3D scene

Patent metadata

Field	Value
Publication number	US-9807365-B2
Application number	US-201514962257-A
Country	US
Kind code	B2
Filing date	Dec 8, 2015
Priority date	Dec 8, 2015
Publication date	Oct 31, 2017
Grant date	Oct 31, 2017

Abstract

A method and system for registering data by first acquiring the data from a scene by a sensor at different viewpoints, and extracting, from the data, three-dimensional (3D) points and 3D lines, and descriptors associated with the 3D points and the 3D lines. A first set of primitives represented in a first coordinate system of the sensor is selected, wherein the first set of primitives includes at least three 3D points. A second set of primitives represented in a second coordinate system is selected, wherein the second set of primitives includes any combination of 3D points and 3D lines to obtain at least three primitives. Then, using the first set of primitives and the second set of primitives, the 3D points are registered with each other, and the 3D points with the 3D lines to obtain registered primitives, wherein the registered primitives are used in a simultaneous localization and mapping (SLAM) system.

First claim

We claim:

1. A method for registering data, comprising steps of:
acquiring the data from a scene by a sensor at different viewpoints;
extracting, from the data, three-dimensional (3D) points and 3D lines, and descriptors associated with the 3D points and the 3D lines, wherein each 3D line is extracted as a ray connecting an optional center of the sensor with a two-dimensional (2) point;
selecting a first set of primitives represented in a first coordinate system of the sensor, wherein the first set of primitives includes at least three 3D points;
selecting a second set of primitives represented in a second coordinate system, wherein the second set of primitives includes any combination of 3D points and 3D lines to obtain at least three primitives; and
registering, using the first set of primitives and the second set of primitives, the 3D points with each other using point-to-point correspondence, and the 3D points with the 3D lines using point-to-line correspondence to obtain registered primitives, wherein the registered primitives are used in a simultaneous localization and mapping (SLAM) system, and wherein the steps are performed in a processor.

2. The method of claim 1 , further comprising:
storing the registered primitives in a SLAM map; and
optimizing the SLAM map by bundle adjustment.

3. The method of claim 1 , wherein the sensor acquires the data as a sequence of frames for the different viewpoints of the scene, wherein each frame includes a pixel image and a corresponding depth map, and the steps are performed for each frame.

4. The method of claim 3 , further comprising:
extracting keypoints from the data, wherein the keypoints include 3D keypoints with valid depths and two-dimensional (2D) keypoints without valid depths;
backprojecting the 3D keypoints with the valid depths to generate the 3D points, and representing the 2D keypoints with the invalid depths as lines passing through the 2D keypoint and an optical center of the sensor based on intrinsic parameters of the sensor to generate the 3D lines.

5. The method of claim 4 , wherein the registering further comprising:
determining correspondences between the first set of primitives and the second set of primitives based on matching descriptors to produce point-to-point, point-to-line, and line-to-line correspondences.

6. The method of claim 1 , wherein the registering is jointly.

7. The method of claim 1 , wherein the data are acquired by multiple sensors.

8. The method of claim 1 , wherein the registering of the 3D points uses a perspective-three-point procedure.

9. The method of claim 1 , wherein the primitivies include 3D planes.

10. The method of claim 9 , wherein registering considers all possible plane-to-plane correspondences.

11. The method of claim 9 , wherein registering is performed via random sample consensus (RANSAC), which is initialized with triplets of correspondences in a following order of 3 plane-to-plane correspondences, 2 plane-to-plane and 1 point-to-point correspondences, 1 plane-to-plane and 2 point-to-point correspondences, 3 point-to-point correspondences, and 3 point-to-line correspondences.

12. A system for registering data, comprising:
a sensor, at different viewpoints, configured to acquiring the data from a scene; and
processor configured to extract, from the data, three-dimensional (3D) points and 3D lines, and descriptors associated with the 3D points and the 3D lines wherein each 3D line is extracted as a ray connecting an optical center of the sensor with a two-dimensional (2D) point, to select a first set of primitives represented in a first coordinate system of the sensor, wherein the first set of primitives includes at least three 3D points, to select a second set of primitives represented in a second coordinate system, wherein the second set of primitives includes any combination of 3D points and 3D lines to obtain at least three primitives, and to register, using the first set of primitives and the second set of primitives, the 3D points with each other using point-to-point correspondence, and the 3D points with the 3D lines using point-to-line correspondence to obtain registered primitives, wherein the registered primitives are used in a simultaneous localization and mapping (SLAM) system.

13. A method for registering data, comprising steps of:
acquiring the data from a scene by a sensor at different viewpoints;
extracting, from the data, three-dimensional (3D) points and 3D lines, and descriptors associated with the 3D points and the 3D lines;
selecting a first set of primitives represented in a first coordinate system of the sensor, wherein the first set of primitives includes at least three 3D points;
selecting a second set of primitives represented in a second coordinate system, wherein the second set of primitives includes any combination of 3D points and 3D lines to obtain at least three primitives; and
registering, using the first set of primitives and the second set of primitives, the 3D points with each other, and the 3D points with the 3D lines to obtain registered primitives, wherein the registered primitives are used in a simultaneous localization and mapping (SLAM) system, wherein the primitives include 3D planes, wherein registering is performed via random sample consensus (RANSAC), which is initialized with triplets of correspondences in a following order of 3 plane-to-plane correspondences, 2 plane-to-plane and 1 point-to-point correspondences, 1 point-to-plane and 2 point-to-point correspondences, 3 point-to-point correspondences, and 3 point-to-point correspondences, and wherein the steps are performed in a processor.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Cansizoglu Esra
• Taguchi Yuichi
• Ramalingam Srikumar

Classifications

• G06T2207/10012

  Stereo images · CPC title

• G06T7/593

  from stereo images · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N13/207 Primary

  using a single two-dimensional [2D] image sensor · CPC title

• G06T2200/04

  involving 3D image data · CPC title