Method and system for generating visual feature map

Patent metadata

Field	Value
Publication number	US-12450772-B2
Application number	US-202218070256-A
Country	US
Kind code	B2
Filing date	Nov 28, 2022
Priority date	May 28, 2020
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

A method for generating a visual feature map includes receiving a first image, detecting a first set of keypoints from the first image, extracting a visual feature descriptor of each of the first set of keypoints, receiving a first set of 3D point cloud data associated with a location where the first image is captured, and determining a 3D coordinate value of each of the first set of keypoints using the first set of 3D point cloud data.

First claim

The invention claimed is:

1. A method for generating a visual feature map performed by one or more processors, said method comprising:
receiving a first image;
detecting a first set of keypoints from the first image;
extracting a visual feature descriptor of each of the first set of keypoints;
receiving a first set of 3D point cloud data associated with a location where the first image is captured;
determining a 3D coordinate value of each of the first set of keypoints using the first set of 3D point cloud data;
receiving a second image;
detecting a second set of keypoints from the second image;
extracting a visual feature descriptor of each of the second set of keypoints;
receiving a second set of 3D point cloud data associated with a location where the second image is captured;
determining a 3D coordinate value of each of the second set of keypoints using the second set of 3D point cloud data;
performing first feature matching by comparing the visual feature descriptors of the first set of keypoints with the visual feature descriptors of the second set of keypoints; and
generating a local map including keypoints, of the second set of keypoints, which succeeded the first feature matching;
wherein the generating of the local map includes:
receiving image capturing direction information of the second image; and
associating the visual feature descriptors, the 3D coordinate values, and the image capturing direction information for each of the keypoints, of the second set of keypoints, which succeeded the first feature matching.

2. The method according to claim 1 , wherein the receiving of the first set of 3D point cloud data includes:
acquiring geometric information of a surface of a road around the location where the first image is captured using a 2D light detection sensor and a ranging (LiDAR) sensor.

3. The method according to claim 2 , wherein the receiving of the first set of 3D point cloud data includes:
acquiring geometric information of a non-road area around the location where the first image is captured using a 3D LiDAR sensor.

4. The method according to claim 1 , wherein the detecting of the first set of keypoints from the first image includes:
detecting a plurality of keypoints from the first image;
identifying, among the plurality of keypoints, keypoints located within a preset distance from a location where the first image was captured; and
determining the identified keypoints as the first set of keypoints.

5. The method according to claim 1 , wherein the detecting of the first set of keypoints from the first image includes:
detecting a plurality of keypoints from the first image;
calculating a distinctiveness score of each of the plurality of keypoints;
identifying, among the plurality of keypoints, keypoints having a distinctiveness score equal to or greater than a preset threshold; and
determining the identified keypoints as the first set of keypoints.

6. The method according to claim 1 , further comprising:
receiving a third image;
detecting a third set of keypoints from the third image;
extracting a visual feature descriptor of each of the third set of keypoints;
receiving a third set of 3D point cloud data associated with a location where the third image is captured;
determining a 3D coordinate value of each of the third set of keypoints using the third set of 3D point cloud data;
performing second feature matching by comparing the visual feature descriptors of the third set of keypoints with the visual feature descriptors of the keypoints in the local map; and
updating the local map based on keypoints, of the third set of keypoints, which succeeded the second feature matching.

7. The method according to claim 6 , wherein the updating of the local map includes:
calculating an average value of visual feature descriptors of a pair of keypoints that succeeded the second feature matching.

8. The method according to claim 6 , further comprising:
performing third feature matching by comparing the visual feature descriptors of keypoints, of the third set of keypoints, which failed the second feature matching with the visual feature descriptors of the second set of keypoints; and
adding keypoints that succeeded the third feature matching to the local map.

9. A non-transitory computer-readable recording medium storing instructions that, when executed by one or more processors, cause performance of the method according to claim 1 .

10. A method for generating a visual feature map performed by one or more processors, said method comprising:
receiving a first image;
detecting a first set of keypoints from the first image;
extracting a visual feature descriptor of each of the first set of keypoints;
receiving a first set of 3D point cloud data associated with a location where the first image is captured;
determining a 3D coordinate value of each of the first set of keypoints using the first set of 3D point cloud data;
receiving a second image;
detecting a second set of keypoints from the second image;
extracting a visual feature descriptor of each of the second set of keypoints;
receiving a second set of 3D point cloud data associated with a location where the second image is captured;
determining a 3D coordinate value of each of the second set of keypoints using the second set of 3D point cloud data;
performing first feature matching by comparing the visual feature descriptors of the first set of keypoints with the visual feature descriptors of the second set of keypoints;
generating a local map including keypoints, of the second set of keypoints, which succeeded the first feature matching;
receiving a third image;
detecting a third set of keypoints from the third image;
extracting a visual feature descriptor of each of the third set of keypoints;
receiving a third set of 3D point cloud data associated with a location where the third image is captured;
determining a 3D coordinate value of each of the third set of keypoints using the third set of 3D point cloud data;
performing second feature matching by comparing the visual feature descriptors of the third set of keypoints with the visual feature descriptors of the keypoints in the local map;
updating the local map based on keypoints, of the third set of keypoints, which succeeded the second feature matching; and
removing from the local map keypoints, of the keypoints included in the local map, which failed the feature matching a predetermined number of times or more.

11. A method for generating a visual feature map performed by one or more processors, said method comprising:
receiving a first image;
detecting a first set of keypoints from the first image;
extracting a visual feature descriptor of each of the first set of keypoints;
receiving a first set of 3D point cloud data associated with a location where the first image is captured;
determining a 3D coordinate value of each of the first set of keypoints using the first set of 3D point cloud data;

receiving a second image;
detecting a second set of keypoints from the second image;
extracting a visual feature descriptor of each of the second set of keypoints;
receiving a second set of 3D point cloud data associated with a location where the second image is captured;
determining a 3D coordinate value of each of the second set of keypoints using the second set of 3D point cloud data;
performing first feature matching by comparing the visual feature descriptors of the first set of keypoints with the visual feature descriptors of the second set of keypoints;
generating a local map including keypoints, of the second set of keypoints, which succeeded the first feature matching;
receivin

Assignees

• Naver Labs Corp
• Naver Corp

Inventors

• Jeon Junho

Classifications

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title