Method and apparatus for performing image feature matching using labeled keyframes in SLAM-based camera tracking

Patent metadata

Field	Value
Publication number	US-10970874-B2
Application number	US-201916370550-A
Country	US
Kind code	B2
Filing date	Mar 29, 2019
Priority date	Mar 30, 2018
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

The present invention relates to a method and an apparatus for performing image feature matching in a SLAM-based camera and, more particularly, to a method and an apparatus for improving the performance and speed of feature matching by generating labeled keyframes including GPS information, receiving limited promising keyframes on the basis of the labeled keyframes, and applying weighted grid to perform image feature matching.

First claim

The invention claimed is:

1. A method of performing image feature matching in a Simultaneous Localization and Mapping (SLAM)-based camera, the method comprising:
processing computer executable program code embodied in non-transitory computer readable storage media by one or more processors, the computer executable program code comprising instructions for:
generating labeled keyframes including Global Positioning System (GPS) information and first information from an input image;
transmitting the labeled keyframes to an Augmented Reality (AR) Apps server;
receiving limited promising keyframes on the basis of the labeled keyframes from the AR Apps server; and
performing image feature matching between the promising keyframes and the input image.

2. The method of claim 1 , wherein the first information included in the labeled keyframes includes a feature descriptor, and the first information further includes at least one of user tag (UTag) information and object tag (OTag) information.

3. The method of claim 1 , wherein the AR app server generates the promising keyframes through keyframe filtering, and the promising keyframes are generated on the basis of any one of the GPS information, the user tag (UTag) included in the first information, and the object tag (OTag) information included in the first information.

4. The method of claim 1 , further comprising:
calculating a difference between latitude and longitude of GPS information received by a terminal and latitude and longitude of the GPS information of the labeled keyframes; and
selecting only labeled keyframes for which the calculated difference is less than a threshold value and using the selected labeled keyframes for the image feature matching.

5. The method of claim 1 , wherein the generating of the labeled keyframes includes:
identifying the GPS information by exchanging information with a peripheral server and database; and
adding the identified GPS information to the labeled keyframes.

6. The method of claim 1 , wherein the first information includes weighted grid information, and the weighted grid is applied to perform the image feature matching.

7. The method of claim 6 , wherein a size of a grid cell to which the weighted grid is applied depends on a size of image resolution.

8. The method of claim 6 , wherein when applying the weighted grid to a first object, the first object is selected by combining movement information of objects in the image and change information of the objects.

9. The method of claim 6 , wherein a value of the weighted grid is adjusted using an AR marker assigned to the input image.

10. The method of claim 6 , further comprising instructions for:
applying the weighted grid to a correspondence relationship between keypoints in a camera input image and keypoints in the keyframes, thereby calculating a distance between corresponding points; and
when the calculated distance is less than a threshold value, determining that the feature matching is performed and thus calculating a camera pose.

11. The method of claim 6 , wherein one or more grid masks to which the weighted grid is applied are used.

12. The method of claim 1 , wherein the image feature matching is performed at high speed by using a soft grid mask.

13. The method of claim 12 , further comprising: when the soft grid mask is applied,
wherein setting values for areas around the weighted grid to zero, and
wherein the image feature matching is performed by correcting weight values in the grid.

14. A terminal performing image feature matching in a Simultaneous Localization and Mapping (SLAM)-based camera, the terminal comprising:
a transceiver; and
a processor controlling the transceiver,
wherein the processor:
generates labeled keyframes including Global Positioning System (GPS) information and first information from an input image;
transmits the labeled keyframes to an Augmented Reality (AR) Apps server;
receives limited promising keyframes on the basis of the labeled keyframes from the AR Apps server; and
performs image feature matching between the promising keyframes and the input image.

15. The terminal of claim 14 , wherein the first information includes weighted grid information, and the weighted grid is applied to perform the image feature matching.

16. The terminal of claim 15 , wherein one or more grid masks to which the weighted grid is applied are used.

17. The terminal of claim 15 , wherein a soft grid mask is used to perform the image feature matching at high speed.

18. An apparatus for performing image feature matching in a Simultaneous Localization and Mapping (SLAM)-based camera, the apparatus comprising:
a terminal generating labeled keyframes including GPS information and first information from an input image;
an AR Apps server in which Augmented Reality (AR) map and keyframe information are stored; and
a computing server inferring the Global Positioning System (GPS) information and recognizing objects in the image using an algorithm;
in which the first information includes weighted grid information, and the weighted grid is applied when performing the image feature matching.

19. The apparatus of claim 18 ,
wherein the AR Apps server generates promising keyframes through keyframe filtering, and
the promising keyframes are generated on the basis of any one of the GPS information, a user tag (UTag) information included in the first information, and an object tag (OTag) information included in the first information.

20. The apparatus of claim 18 ,
wherein the computing server identifies the GPS information by exchanging information with a peripheral server and database; and
adds the identified GPS information to the labeled keyframes.

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Bang Jun Seong

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/757

  Matching configurations of points or features · CPC title

• G06V10/225

  based on a marking or identifier characterising the area · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title