Method and System for Concurrent Reconstruction of Dynamic and Static Objects

Patent metadata

Field	Value
Publication number	US-2019050668-A1
Application number	US-201715674843-A
Country	US
Kind code	A1
Filing date	Aug 11, 2017
Priority date	Aug 11, 2017
Publication date	Feb 14, 2019
Grant date	—

Abstract

An imaging system for localization and mapping of a scene including static and dynamic objects. A sensor acquires a sequence of frames in motion or stationary. A memory to store a static map of static objects and an object map of each dynamic object in the scene. The static map includes a set of landmarks, and the object map includes a set of landmarks and a set of segments. A localizer registers keypoints of the frame with landmarks in the static map using frame-based registration and to register some segments in the frame with segments in the object map using a segment-based registration. A mapper to update each object map with keypoints forming each segment and keypoints registered with the corresponding object map according to the segment-based registration, and to update the static map with the remaining keypoints in the frame using the keypoints registered with the static map.

First claim

What is claimed is:

1 . An imaging system for localization and mapping of a scene including static and dynamic objects, comprising:
a sensor to acquire a sequence of frames, wherein each frame includes a set of keypoints representing intensity and depth measurements of the scene; a memory to store a static map of one or multiple static objects in the scene and an object map of each dynamic object in the scene, wherein the static map includes a set of landmarks, and wherein the object map includes a set of landmarks and a set of segments, wherein a descriptor of each segment is a function of multiple measurements forming the segment; a localizer to register at least some keypoints of the frame acquired by the sensor with landmarks in the static map using frame-based registration, and to register at least some segments in the frame with segments in the object map using a segment-based registration; and a mapper to update each object map with keypoints forming each segment and keypoints registered with the corresponding object map according to the segment-based registration, and to update the static map with the remaining keypoints in the frame using the keypoints registered with the static map.

2 . The imaging system of claim 1 , wherein the localizer is configured to extract features from the frame;
segment the frame into multiple segments, wherein each segment includes multiple features extracted from the frame; search the object map for segments similar to the segments in the frame; and register a segment in the frame with a matching segment in the object map.

3 . The imaging system of claim 2 , wherein the search is based on a vector of locally aggregated descriptors (VLAD).

4 . The imaging system of claim 2 , wherein the frame is segmented with a depth-based segmentation.

5 . The imaging system of claim 2 , wherein the features are associated with descriptors.

6 . The imaging system of claim 2 , wherein the landmarks correspond to specific discriminative points on an object in the scene.

7 . The imaging system of claim 2 , wherein the segment is registered using a random sample consensus (RANSAC).

8 . The imaging system of claim 2 , wherein the search is an appearance-based similarity search.

9 . The imaging system of claim 1 , wherein the segment-based registration performs appearance matching of segments in the frame with the segments in the object map and performs a geometric verification between the matching segments.

10 . The imaging system of claim 1 , wherein the mapper is configured to add all keypoints of one or more segment to the object map that the segment is registered.

11 . The imaging system of claim 1 , wherein the mapper is further configured to add keypoints to the object map based on the localization, if the keypoints have a correspondence with the landmarks of the object map, if not, add the keypoints to the static map.

12 . The imaging system of claim 1 , further comprising:
a manipulator including a robot arm for manipulating the dynamic object based on the object map.

13 . The imaging system of claim 1 , wherein the sensor acquires the sequence of frames while in motion or stationary, and is from the group consisting one of a three dimensional (3D) sensor capable of acquiring a red color, a green color, a blue color, and a depth (RGB-D) image data, or other sensors capable of acquiring image data that is composed of intensity and depth.

14 . An imaging system for localization and mapping of a scene including static and dynamic objects, wherein a sensor acquires sensor data while in motion, such that the data includes a sequence of frames and each frame includes a set of keypoints representing intensity and depth measurements of the scene, and a memory stores a static map of one or multiple static objects in the scene and an object map of each dynamic object in the scene, wherein the static map includes a set of landmarks, and wherein the object map includes a set of segments, such that a descriptor of each segment is a function of multiple measurements forming the segment, the imaging system comprising:
a processor in communication with the memory, is configured to employ a localizer to register at least some keypoints of the frame acquired by the sensor with landmarks in the static map using frame-based registration and to register at least some segments in the frame with segments in the object map using a segment-based registration, wherein the landmarks correspond to specific discriminative points on an object in the scene; and a mapper to update each object map with keypoints forming each segment and keypoints registered with the corresponding object map according to the segment-based registration, and to update the static map with the remaining keypoints in the frame using the keypoints registered with the static map.

15 . The imaging system of claim 14 , wherein the localizer is configured to extract features from the frame;
segment the frame into multiple segments, wherein each segment includes multiple features extracted from the frame; search the object map for segments similar to the segments in the frame; and register a segment in the frame with a matching segment in the object map.

16 . The imaging system of claim 14 , wherein the mapper is configured to add all keypoints of one or more segment to the object map that the segment is registered.

17 . The imaging system of claim 15 , wherein the mapper is further configured to add keypoints to that object map based on the localization, if the keypoints have a correspondence with the landmarks of the object map, if not, add the keypoints to the static map.

18 . An imaging system for localization and mapping of a scene including static and dynamic objects, comprising:
a sensor to acquire sensor data while in motion or stationary, such that the sensor data includes a sequence of frames, wherein each frame includes a set of keypoints representing intensity and depth measurements of the scene; a memory to store a static map of one or multiple static objects in the scene and an object map of each dynamic object in the scene via an input interface, wherein the static map includes a set of landmarks, and wherein the object map includes a set of segments, such that a descriptor of each segment is a function of multiple measurements forming the segment, wherein the measurements of the frame further include at least 3D planes; a processor in communication with the sensor, memory and input interface, the processor is configured to employ a localizer to register at least some keypoints of the frame acquired by the sensor with landmarks in the static map using frame-based registration and to register at least some segments in the frame with segments in the object map using a segment-based registration; and a mapper to update each object map with keypoints forming each segment and keypoints registered with the corresponding object map according to the segment-based registration, and to update the static map with the remaining keypoints in the frame using the keypoints registered with the static map.

19 . The imaging system of claim 18 , wherein the localizer is configured to extract features from the frame;
segment the frame into multiple segments, wherein each segment includes multiple features extracted from the frame; search the object map for segments similar to the segments in the frame; and register a segment in the frame with a matching segment in the object map.

20 . The imaging syst

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Cansizoglu Esra
• Caccamo Sergio S
• Taguchi Yuichi

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G05D1/0274

  using mapping information stored in a memory device (navigation using map-matching G01C21/30) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/55

  from multiple images · CPC title

• G06T7/337

  involving reference images or patches · CPC title