Systems and methods for image retrieval using super features

Patent metadata

Field	Value
Publication number	US-12164559-B2
Application number	US-202217581158-A
Country	US
Kind code	B2
Filing date	Jan 21, 2022
Priority date	Oct 5, 2021
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

Am image retrieval system includes: a neural network (NN) module configured to generate local features based on an input image; an iterative attention module configured to, via T iterations, generate an ordered set of super features in the input image based on the local features, where T is an integer greater than 1; and a selection module configured to select a second image from a plurality of images in an image database based on the second image having a second ordered set of super features that most closely match the ordered set of super features in the input image, where the super features in the set of super features do not include redundant local features of the input image.

First claim

What is claimed is:

1. An image retrieval system, comprising:
a neural network (NN) module configured to generate local features based on an input image;
an iterative attention module configured to, via T iterations, generate an ordered set of super features in the input image based on the local features,
where T is an integer greater than 1; and
a selection module configured to select a second image from a plurality of images in an image database based on the second image having a second ordered set of super features that most closely match the ordered set of super features in the input image,
wherein the super features in the set of super features do not include redundant local features of the input image.

2. The image retrieval system of claim 1 wherein the iterative attention module is configured to:
during a first one of the T iterations, generate a third ordered set of super features in the input image further based on an ordered set of predetermined initialization super features and the local features; and
generate the ordered set of super features based on the third ordered set of super features.

3. The image retrieval system of claim 2 wherein the iterative attention module is configured to:
during a second one of the T iterations that is after the first one of the T iterations, generate a fourth ordered set of super features in the input image further based on the third ordered set of super features and the local features; and
generate the ordered set of super features based on the fourth ordered set of super features.

4. The image retrieval system of claim 1 wherein the iterative attention module is configured to:
during an N-th one of the T iterations, where N is an integer less than or equal to T, generate an N-th ordered set of super features in the input image further based on (a) an N−1th ordered set of super features for an N−1th iteration of the T iterations and (b) the local features; and
generate the ordered set of super features based on the N-th ordered set of super features.

5. The image retrieval system of claim 4 wherein the iterative attention module is configured to, during the N-th one of the T iterations:
determine a first linear projection of the local features;
determine a second linear projection of the local features;
determine a third linear projection of the N−1th ordered set of super features; and
generate the ordered set of super features in the input image based on the first, second, and third linear projections.

6. The image retrieval system of claim 5 wherein the iterative attention module is configured to:
determine a fourth linear projection based on a product of the first and third linear projections;
generate attention maps for the input image based on the fourth linear projection; and
generate the ordered set of super features in the input image based on the attention maps and the second linear projection.

7. The image retrieval system of claim 6 wherein the iterative attention module is configured to generate the attention maps for the input image by:
scaling the fourth linear projection;
applying a softmax function after the scaling; and
normalizing a result of the scaling.

8. The image retrieval system of claim 6 wherein the iterative attention module is configured to:
determine first features based on the attention maps and the second linear projection;
determine second features based on the third linear projection and the first features; and
generate the ordered set of super features in the input image based on the second features.

9. The image retrieval system of claim 8 wherein the iterative attention module includes a multi layer perceptron (MLP) module configured to generate an output based on the second features,
wherein the iterative attention module is configured to generate the ordered set of super features in the input image based on (a) the second features and (b) the output of the MLP module.

10. The image retrieval system of claim 1 wherein the iterative attention module is trained by minimizing a contrastive loss.

11. The image retrieval system of claim 1 wherein the iterative attention module is trained by minimizing a cosine similarity loss.

12. The image retrieval system of claim 1 wherein:
the NN module receives the input image from a computing device via a network; and
the selection module transmits the second image to the computing device via the network.

13. The image retrieval system of claim 12 further comprising the computing device, wherein the computing device is configured to at least one of:
display the second image on a display; and
display information regarding the second image on the display; and
audibly output the information regarding the second image via a speaker.

14. A system, comprising:
the image retrieval system of claim 1 ;
a camera configured to capture the input image; and
a pose and location module configured to determine at least one of:
a present location based on the second image; and
a pose of the camera based on the second image.

15. An image retrieval method, comprising:
by a neural network (NN) module, generating local features based on an input image;
via T iterations, generating an ordered set of super features in the input image based on the local features using an attention module,
where T is an integer greater than 1; and
selecting a second image from a plurality of images in an image database based on the second image having a second ordered set of super features that most closely match the ordered set of super features in the input image,
wherein the super features in the set of super features do not include redundant local features of the input image.

16. The image retrieval method of claim 15 wherein the generating the ordered set of super features includes:
during a first one of the T iterations, generating a third ordered set of super features in the input image further based on an ordered set of predetermined initialization super features and the local features; and
generating the ordered set of super features based on the third ordered set of super features.

17. The image retrieval method of claim 16 wherein the generating the ordered set of super features includes:
during a second one of the T iterations that is after the first one of the T iterations, generating a fourth ordered set of super features in the input image further based on the third ordered set of super features and the local features; and
generating the ordered set of super features based on the fourth ordered set of super features.

18. The image retrieval method of claim 15 wherein the generating the ordered set of super features includes:
during an N-th one of the T iterations, where N is an integer less than or equal to T, generating an N-th ordered set of super features in the input image further based on (a) an N−1th ordered set of super features for an N−1th iteration of the T iterations and (b) the local features; and
generating the ordered set of super features based on the N-th ordered set of super features.

19. The image retrieval method of claim 18 wherein the generating the ordered set of super features includes, during the N-th one of the T iterations:
determining a first linear projection of the local features;
determining a second linear projection of the local features;
determining a third linear projection of the N−1th ordered set of super features; and
generating the ordered set of super features in the input image based on the first, second,

Assignees

• Naver Corp

Inventors

• Weinzaepfel Philippe
• Lucas Thomas
• Larlus Diane
• Kalantidis Ioannis

Classifications

• G06N3/08

  Learning methods · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title