Learning device, search device, learning method, search method, learning program, and search program

Patent metadata

Field	Value
Publication number	US-11790635-B2
Application number	US-201917619239-A
Country	US
Kind code	B2
Filing date	Jun 17, 2019
Priority date	Jun 17, 2019
Publication date	Oct 17, 2023
Grant date	Oct 17, 2023

Abstract

A retrieval apparatus includes a first retrieval unit, a second retrieval unit, and an integration unit that calculates an integrated similarity by integrating a first similarity calculated by the first retrieval unit and a second similarity calculated by the second retrieval unit. For similarities between a basic image, an image similar to the basic image, and an image dissimilar to the basic image of the reference images, at least the feature extraction of the first retrieval unit is learned such that a margin based on a second similarity between the basic image and the similar image and a second similarity between the basic image and the dissimilar image increases as the second similarity between the basic image and the dissimilar image increases relative to the second similarity between the basic image and the similar image.

First claim

The invention claimed is:

1. A retrieval apparatus comprising circuitry configured to execute operations comprising:
receiving a query image to be retrieve;
calculating a first similarity as a similarity between feature vectors, the first similarity being calculated between a feature vector extracted from the query image through learned feature extraction for outputting the feature vector and a feature vector extracted from reference images labeled by the feature extraction;
calculating a second similarity as a similarity determined using information on features, based on information on a feature different from the feature vector of the query image and information on features of the reference images; and
calculating an integrated similarity by integrating the first similarity and the second similarity, wherein
for similarities between a basic image, an image similar to the basic image, and a dissimilar image that is dissimilar to the basic image of the reference images, at least the feature extraction is learned such that a margin based on a second similarity between the basic image and the similar image and a second similarity between the basic image and the dissimilar image increases as the second similarity between the basic image and the dissimilar image increases relative to the second similarity between the basic image and the similar image.

2. A learning apparatus comprising circuitry configured to execute operations comprising:
calculating a second similarity for a combination of reference images by using information on features different from feature vectors of labeled reference images, the second similarity being calculated as a similarity determined using the information on the features,
the reference images including a basic image as a reference of the labeling, a similar image that is the reference image similar to the basic image, and a dissimilar image that is the reference image dissimilar to the basic image; and

updating a parameter of a neural network such that a margin increases as the second similarity between the basic image and the dissimilar image increases relative to a second similarity between the basic image and the similar image, the parameter being updated by using a loss function including a first similarity between a feature vector of the basic image and a feature vector of the similar image, a first similarity between the feature vector of the basic image and a feature vector of the dissimilar image, and the margin based on the second similarity between the basic image and the similar image and the second similarity between the basic image and the dissimilar image, the feature vectors being outputted from the neural network for receiving a predetermined image and outputting the feature vectors.

3. The learning apparatus according to claim 2 , wherein for a set of the basic image, the similar image, and the dissimilar image, the circuitry further configured to execute a method comprising: updating the parameter of the neural network according to the loss function by using only the set of the three images if the second similarity between the basic image and the dissimilar image is at least a threshold value relative to the second similarity between the basic image and the similar image and the margin is at least a threshold value.

4. A computer-implemented retrieval method for retrieving, comprising:
receiving a query image to be retrieved;
calculating a first similarity as a similarity between feature vectors, the first similarity being calculated between a feature vector extracted from the query image through learned feature extraction for outputting the feature vector and a feature vector extracted from reference images labeled by the feature extraction;
calculating a second similarity as a similarity determined using information on features, based on information on a feature different from the feature vector of the query image and information on the features of the reference images; and
calculating an integrated similarity by integrating the calculated first similarity and the calculated second similarity, wherein
for similarities between a basic image, an image similar to the basic image, and a dissimilar image that is dissimilar to the basic image of the reference images, at least the feature extraction is learned such that a margin based on a second similarity between the basic image and the similar image and a second similarity between the basic image and the dissimilar image increases as the second similarity between the basic image and the dissimilar image increases relative to the second similarity between the basic image and the similar image.

5. The retrieval apparatus according to claim 1 , the circuitry further configured to execute a method comprising:
determining an identity of the query image based on the first similarity between the feature vector extracted from a region in the query image and another feature vector extracted from another region that corresponds to the region in a reference image, wherein the feature vector indicates a local feature amount of the query image.

6. The retrieval apparatus according to claim 1 , wherein the first similarity is based on a cosine similarity between the feature vector extracted from the query image through learned feature extraction and the feature vector extracted from a reference image of the reference images labeled by a feature extraction.

7. The retrieval apparatus according to claim 1 , wherein the second similarity is based on a similarity between a first local feature amount associated with a first feature associated with the feature vector extracted from the query image through learned feature extraction and a second local feature amount associated with a second feature associated with the feature vector extracted from the reference images labeled by a feature extraction.

8. The learning apparatus according to claim 2 , wherein the first similarity is based on a cosine similarity between a first feature vector associated with the basic image and a second feature vector associated with a reference image.

9. The learning apparatus according to claim 2 , wherein the second similarity is based on a similarity between a first local feature amount associated with a first feature associated with the feature vector for the basic image and a second local feature amount associated with a second feature for a reference image.

10. The computer-implemented method according to claim 4 , the method further comprising:
determining an identity of the query image based on the first similarity between the feature vector extracted from a region in the query image and another feature vector extracted from another region that corresponds to the region in a reference image, wherein the feature vector indicates a local feature amount of the query image.

11. The computer-implemented method according to claim 4 , wherein the first similarity is based on a cosine similarity between the feature vector extracted from the query image through learned feature extraction and the feature vector extracted from a reference image labeled by a feature extraction.

12. The computer-implemented method according to claim 4 , wherein the second similarity is based on a similarity between a first local feature amount associated with a first feature associated with the feature vector extracted from the query image through learned feature extraction and a second local feature amount associated with a second feature associated with the feature vector extracted from the reference images labeled by the feature extraction.

Assignees

• Nippon Telegraph & Telephone

Inventors

• Watanabe Yukito
• Umeda Takayuki
• Shimamura Jun
• Sagata Atsushi

Classifications

• G06V10/761 Primary

  Proximity, similarity or dissimilarity measures · CPC title

• G06F16/532 Primary

  Query formulation, e.g. graphical querying · CPC title

• G06F16/56

  having vectorial format · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G06V10/778

  Active pattern-learning, e.g. online learning of image or video features · CPC title