Feature descriptor matching

Patent metadata

Field	Value
Publication number	US-10997746-B2
Application number	US-201916381373-A
Country	US
Kind code	B2
Filing date	Apr 11, 2019
Priority date	Apr 12, 2018
Publication date	May 4, 2021
Grant date	May 4, 2021

Abstract

Feature descriptor matching described herein may include receiving a first input image and a second input image. A feature detector may detect features from the first and second input images. A descriptor extractor may learn local feature descriptors from the features of the first and second input images based on a feature descriptor matching model trained using a ground truth data set. The descriptor extractor may determine a listwise mean average precision (mAP) rank of a pool of candidate image patches from the second input image with respect to a queried image patch from the first input image based on the feature descriptor matching model, the first set of local feature descriptors, and the second set of local feature descriptors. The descriptor matcher may generate a geometric transformation between the first input image and the second input image based on the listwise mAP and a convolutional neural network.

First claim

The invention claimed is:

1. A system for feature descriptor matching, comprising:
a memory for receiving a first input image and a second input image;
a feature detector for detecting a first set of features from the first input image and a second set of features from the second input image;
a descriptor extractor for learning a first set of local feature descriptors from the first set of features of the first input image and a second set of local feature descriptors from the second set of features of the second input image based on a feature descriptor matching model trained using a ground truth data set including a first ground truth image and a second ground truth image;
the descriptor extractor for determining a listwise mean average precision (mAP) rank of a pool of candidate image patches from the second input image with respect to a queried image patch from the first input image based on the feature descriptor matching model, the first set of local feature descriptors, and the second set of local feature descriptors; and
a descriptor matcher for generating a geometric transformation between the first input image and the second input image based on the listwise mAP and a convolutional neural network (CNN),
wherein a geometric alignment is performed on the first ground truth image and the second ground truth image prior to training the feature descriptor matching model using the first ground truth image and the second ground truth image.

2. The system for feature descriptor matching of claim 1 , wherein the first set of local feature descriptors or the second set of local feature descriptors includes a vector representation of a corresponding image patch.

3. The system for feature descriptor matching of claim 1 , wherein the first set of local feature descriptors or the second set of local feature descriptors includes a binary descriptor or a real-valued descriptor.

4. The system for feature descriptor matching of claim 3 , wherein the descriptor matcher generates the geometric transformation further based on an amount of computing resources available within the system for feature descriptor matching and one of:
binary descriptors of the first set of local feature descriptors and binary descriptors of the second set of local feature descriptors; or
real-valued descriptors of the first set of local feature descriptors and real-valued descriptors of the second set of local feature descriptors.

5. The system for feature descriptor matching of claim 1 , wherein the first ground truth image and the second ground truth image of the ground truth data set are hard negatives with no matching features.

6. The system for feature descriptor matching of claim 1 , wherein the geometric alignment is performed based on a spatial transformer network.

7. The system for feature descriptor matching of claim 1 , wherein the descriptor extractor performs label mining based on clustering while learning the first set of local feature descriptors or the second set of local feature descriptors.

8. The system for feature descriptor matching of claim 7 , wherein one or more image patches are clustered based on an inter-cluster distance from other image patches.

9. The system for feature descriptor matching of claim 1 , wherein the feature descriptor matching model or the CNN is trained based on stochastic gradient descent (SGD).

10. A method for feature descriptor matching, comprising:
receiving a first input image and a second input image;
detecting a first set of features from the first input image and a second set of features from the second input image;
learning a first set of local feature descriptors from the first set of features of the first input image and a second set of local feature descriptors from the second set of features of the second input image based on a feature descriptor matching model trained using a ground truth data set including a first ground truth image and a second ground truth image;
determining a listwise mean average precision (mAP) rank of a pool of candidate image patches from the second input image with respect to a queried image patch from the first input image based on the feature descriptor matching model, the first set of local feature descriptors, and the second set of local feature descriptors;
generating a geometric transformation between the first input image and the second input image based on the listwise mAP and a convolutional neural network (CNN); and
performing a geometric alignment on the first ground truth image and the second ground truth image prior to training the feature descriptor matching model using the first ground truth image and the second ground truth image.

11. The method for feature descriptor matching of claim 10 , wherein the first set of local feature descriptors or the second set of local feature descriptors includes a binary descriptor or a real-valued descriptor.

12. The method for feature descriptor matching of claim 11 , further comprising generating the geometric transformation further based on an amount of computing resources available within a system for feature descriptor matching and one of:
binary descriptors of the first set of local feature descriptors and binary descriptors of the second set of local feature descriptors; or
real-valued descriptors of the first set of local feature descriptors and real-valued descriptors of the second set of local feature descriptors.

13. The method for feature descriptor matching of claim 10 , wherein the first ground truth image and the second ground truth image of the ground truth data set are hard negatives with no matching features.

14. The method for feature descriptor matching of claim 10 , wherein the geometric alignment is performed based on a spatial transformer network.

15. A system for feature descriptor matching, comprising:
a memory for receiving a first input image and a second input image;
a feature detector for detecting a first set of features from the first input image and a second set of features from the second input image;
a descriptor extractor for learning a first set of local feature descriptors from the first set of features of the first input image and a second set of local feature descriptors from the second set of features of the second input image based on a feature descriptor matching model trained using a ground truth data set including a first ground truth image and a second ground truth image, wherein the first set of local feature descriptors and the second set of local feature descriptors include a binary descriptor or a real-valued descriptor;
the descriptor extractor for determining a listwise mean average precision (mAP) rank of a pool of candidate image patches from the second input image with respect to a queried image patch from the first input image based on the feature descriptor matching model, the first set of local feature descriptors, and the second set of local feature descriptors; and
a descriptor matcher for generating a geometric transformation between the first input image and the second input image based on the listwise mAP and a convolutional neural network (CNN),
wherein a geometric alignment is performed on the first ground truth image and the second ground truth image prior to training the feature descriptor matching model using the first ground truth image and the second ground truth image.

16. The system for feature descriptor matching of claim 15 , wherein the descriptor matcher generates the geometric transformation further based on an amount of computing resources available within the system for feature descriptor matching and

Assignees

• Honda Motor Co Ltd

Inventors

• He Kun
• Lu Yan

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06T7/337 Primary

  involving reference images or patches · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/50

  by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis · CPC title