Loop closure detection method, mobile device and computer readable storage medium

Patent metadata

Field	Value
Publication number	US-11423646-B2
Application number	US-202016953378-A
Country	US
Kind code	B2
Filing date	Nov 20, 2020
Priority date	Aug 3, 2020
Publication date	Aug 23, 2022
Grant date	Aug 23, 2022

Abstract

A loop closure detection method, a mobile device, and a computer readable storage medium are provided. The method includes: collecting images in different detection directions simultaneously through C0 cameras installed on the mobile device to obtain an image data group comprising C0 images; calculating feature information of each image in the image data group; performing a loop closure detection in C0 sub-threads respectively based on the feature information to obtain a loop closure detection result of each sub-thread; and determining a loop closure detection result of the mobile device based on the loop closure detection result of each sub-thread. In this manner, cross detections in a plurality of detection directions can be realized, which breaks through the limitation of loop closure detection in the prior art with respect to path direction, avoids repeated paths in the same direction, and greatly improves the mapping efficiency.

First claim

What is claimed is:

1. A computer-implemented loop closure detection method for a mobile device, comprising steps of:
collecting images in different detection directions simultaneously through C0 cameras installed on the mobile device to obtain an image data group comprising C0 images, wherein C0 is an integer larger than 1;
calculating feature information of each image in the image data group;
performing a loop closure detection in C0 sub-threads respectively based on the feature information to obtain a loop closure detection result of each sub-thread, wherein each sub-thread corresponds to the loop closure detection in one detection direction; and
determining a loop closure detection result of the mobile device based on the loop closure detection result of each sub-thread;
wherein the step of determining the loop closure detection result of the mobile device based on the loop closure detection result of each sub-thread comprises:
selecting the loop closure detection result with a largest inlier point matching amount from the loop closure detection results of the sub-threads to take as the loop closure detection result of the mobile device.

2. The method of claim 1 , wherein the step of calculating the feature information of each image in the image data group comprises:
extracting a corner point feature set of a j-th image in the image data group, wherein 0≤j<C0;
calculating a descriptor set corresponding to the corner point feature set; and
using the corner point feature set and the descriptor set as feature information of the j-th image in the image data group.

3. The method of claim 1 , wherein the step of performing the loop closure detection in C0 sub-threads respectively based on the feature information to obtain the loop closure detection result of each sub-thread comprises:
traversing the feature information of each image in a i-th sub-thread to obtain the loop closure detection result corresponding to the feature information of each image in the i-th sub-thread, wherein 0≤i<C0; and
determining the loop closure detection result of the i-th sub-thread based on the loop closure detection result corresponding the feature information of each image in the i-th sub-thread.

4. The method of claim 3 , wherein the step of traversing the feature information of each image in the i-th sub-thread to obtain the loop closure detection result corresponding to the feature information of each image in the i-th sub-thread comprises:
inserting the feature information of a j-th image into a target loop closure database, wherein the target loop closure database is a j-th loop closure database of the i-th sub-thread, wherein 0≤j<C0;
determining loop closure query frame information corresponding to the target loop closure database, and performing a loop closure frame information retrieval in the target loop closure database based on the loop closure query frame information;
calculating an inlier point matching amount between the loop closure frame information and the loop closure query frame information, in response to the loop closure frame information being retrieved; and
determining the loop closure detection result corresponding to the feature information of the j-th image in the i-th sub-thread based on the inlier point matching amount.

5. The method of claim 4 , wherein the step of determining the loop closure query frame information corresponding to the target loop closure database comprises:
determining a sequence number of the loop closure query frame information in the target loop closure database based on the following formula:
c =( j+i )% C 0;

where, c is the sequence number of the loop closure query frame information in the target loop closure database, and 0≤c<C0.

6. The method of claim 4 , wherein the step of determining the loop closure detection result corresponding to the feature information of the i-th image in the i-th sub-thread based on the inlier point matching amount, comprises:
in response to the inlier point matching amount being less than a preset matching amount threshold, determining that the loop closure detection result corresponding to the feature information of the j-th image in the i-th sub-thread is null; and
in response to the inlier point matching amount being larger than or equal to the preset matching amount threshold, determining that the loop closure detection is valid, and caching a current frame index, a loop closure frame index, an image number in the current frame, a corresponding loop closure frame image number, and an inlier point matching relationship to take as the loop closure detection result corresponding to the feature information of the j-th image ire the i-th sub-thread.

7. The method of claim 3 , wherein the step of determining the loop closure detection result of the i-th sub-thread based on the loop closure detection result corresponding to the feature information of each image in the i-th sub-thread comprises:
selecting the loop closure detection result with a largest inlier point matching amount from the loop closure detection result corresponding, to the feature information of each image in the i-th sub-thread to take as the loop closure detection result of the i-th sub-thread.

8. The method of claim 1 , wherein the method further comprises:
loading a configuration file of the C0 cameras, reading an amount of the C0 cameras from the configuration file, and creating a main thread and the C0 sub-threads at the same time;
wherein each of the C0 sub-threads is configured to perform a loop closure detection task which corresponds to a loop closure detection in one detection direction; and
wherein the main thread is configured to obtain the image data group comprising the C0 images, calculate the feature information of each image in the Image data group, distribute the calculated feature information to the C0 sub-threads in turn, and determine the loop closure detection result of the mobile device based on the loop closure detection result of each of the C0 sub-threads which is feedbacked by each of the C0 sub-threads.

9. A mobile device, comprising:
C0 cameras;
a memory;
a processor; and
one or more computer programs stored in the memory and executable on the processor, wherein the one or more computer programs comprise:
instructions for collecting images in different detection directions simultaneously through the C0 cameras to obtain an image data group comprising C0 images, wherein C0 is an integer larger than 1;
instructions for calculating feature information of each image in the image data group;
instructions for performing a loop closure detection in C0 sub-threads respectively based on the feature information to obtain a loop closure detection result of each sub-thread, wherein each sub-thread corresponds to the loop closure detection in one detection direction; and
instructions for determining a loop closure detection result of the mobile device based on the loop closure detection result of each sub-thread;
wherein the instructions for determining the loop closure detection result of the mobile device based on the loop closure detection result of each sub-thread comprise:
instructions for selecting the loop closure detection result with a largest inlier point matching amount from the loop closure detection results of the sub-threads to take as the loop closure detection result of the mobile device.

10. The mobile device of claim 9 , wherein the instructions for calculating the feature information of each image in the image data group comprise:
instructions for extracting a corner point feature set of a j-th image in the image data group, wherein 0≤j<C0;
instructions for calculating a descriptor set corresponding to th

Assignees

• Ubtech Robotics Corp Ltd

Inventors

• Guo Rui
• Liu Zhichao
• Jiang Chenchen
• Xiong Youjun

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G01C21/20 Primary

  Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence) · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title