Method and electronic device for retrieving an image and computer readable storage medium

Patent metadata

Field	Value
Publication number	US-11113586-B2
Application number	US-201916531644-A
Country	US
Kind code	B2
Filing date	Aug 5, 2019
Priority date	Jan 29, 2019
Publication date	Sep 7, 2021
Grant date	Sep 7, 2021

Abstract

According to the embodiments of the present application, there are proposed a method and electronic device for retrieving an image, and computer readable storage medium. The method includes: processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved; processing the image to be retrieved using a second neural network to determine a global feature vector of the image to be retrieved; and determining, based on the local feature vector and the global feature vector, an image having a similarity to the image to be retrieved which is higher than a similarity threshold.

First claim

I claim:

1. A method for retrieving an image, comprising:
processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved;
processing the image to be retrieved using a second neural network to determine a global feature vector of the image to be retrieved; and
determining, based on the local feature vector and the global feature vector, an image having a similarity to the image to be retrieved which is higher than a similarity threshold;
wherein the first neural network is trained by using a loss function as follows:
L t ( ya,yp,yn )=max(∥ ya−yp∥ 2 2 −∥ya−yn∥ 2 2 +α,0),

where L t represents a loss function for the first neural network, ya is a feature vector of a standard image, yp is a feature vector of a positive sample, yn is a feature vector of a negative sample, ∥⋅∥ 2 2 represents a square of 2-norm of a vector, max( ) represents a maximum value solving function, and α is margin value.

2. The method according to claim 1 , wherein the first neural network is trained using a plurality of training image data having different resolutions of a training image, and the first neural network is used for processing a plurality of image data to be retrieved having different resolutions of the image to be retrieved.

3. The method according to claim 2 , wherein a number of pixels of the shortest side of the plurality of training image data having different resolutions or the plurality of training image data having different resolutions comprises at least two of 256, 384, 512, 640, and 768.

4. The method according to claim 2 , wherein the first neural network comprises the following convolutional layers:
a first convolutional layer having 96 convolution kernels with a dimension of 11*11*3;
a second convolutional layer having 256 convolution kernels with a dimension of 5*5*96;
a third convolutional layer having 384 convolution kernels with a dimension of 3*3*256;
a fourth convolutional layer having 384 convolution kernels with a dimension of 3*3*384;
a fifth convolutional layer having 256 convolution kernels with a dimension of 3*3*384;
a sixth convolutional layer having 4096 convolution kernels with a dimension of 1*1*256; and
a seventh convolutional layer having 4096 convolution kernels with a dimension of 13*13*4096.

5. The method according to claim 4 , wherein the first neural network further comprises a max pooling layer and a sum pooling layer subsequent to the seventh convolutional layer.

6. The method according to claim 1 , wherein α is defined as:
α=0.5*∥ yp−yn∥ 2 2

7. The method according to claim 4 , wherein the step of processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved comprises:
processing, by using each convolutional layer in the first neural network, a plurality of image data to be retrieved having different resolutions of the image to be retrieved, and determining a plurality of receptive fields respectively having a maximum activation value in a plurality of feature maps for the respective resolutions as an output; and
processing the plurality of receptive fields using a sum pooling layer in the first neural network to determine the local feature vector.

8. The method according to claim 1 , wherein the second neural network comprises the following convolutional layers:
a first convolutional layer having 96 convolution kernels with a dimension of 11*11*3;
a second convolutional layer having 256 convolution kernels with a dimension of 5*5*96;
a third convolutional layer having 384 convolution kernels with a dimension of 3*3*256;
a fourth convolutional layer having 384 convolution kernels with a dimension of 3*3*384;
a fifth convolutional layer having 256 convolution kernels with a dimension of 3*3*384;
a first fully connected layer with a dimension of 1*4096; and
a second fully connected layer with a dimension of 1*4096.

9. The method according to claim 8 , wherein the second neural network further has a spatial transformer network between the fifth convolutional layer and the first fully connected layer.

10. The method according to claim 1 , wherein the loss function used for training the second neural network is a loss function as follows:

L
s

⁡

(

y
⁢
1

,

⁢

y
⁢
2

,

⁢
y

)

=

(

1
-
y

)

2

⁢



y
⁢

⁢
1

-

y
⁢

⁢
2



2
2

+

y
2

⁢

max
⁡

(

m
-



y
⁢

⁢
1

-

y
⁢

⁢
2

Assignees

• Boe Technology Group Co Ltd

Inventors

• Wang Tingting

Classifications

• G06V10/806

  of extracted features · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title