Identifying visually similar digital images utilizing deep learning

Patent metadata

Field	Value
Publication number	US-11227185-B2
Application number	US-202016817234-A
Country	US
Kind code	B2
Filing date	Mar 12, 2020
Priority date	May 18, 2018
Publication date	Jan 18, 2022
Grant date	Jan 18, 2022

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for utilizing a deep neural network-based model to identify similar digital images for query digital images. For example, the disclosed systems utilize a deep neural network-based model to analyze query digital images to generate deep neural network-based representations of the query digital images. In addition, the disclosed systems can generate results of visually-similar digital images for the query digital images based on comparing the deep neural network-based representations with representations of candidate digital images. Furthermore, the disclosed systems can identify visually similar digital images based on user-defined attributes and image masks to emphasize specific attributes or portions of query digital images.

First claim

What is claimed is:

1. A computer-implemented method for identifying digital images based on visual similarity comprising:
receiving a search request comprising a query digital image and an indication of an area of the query digital image to emphasize;
generating a deep neural network-based representation of the query digital image that applies a first weight to deep features corresponding to the area of the query digital image to emphasize and applies a second weight to deep features corresponding to portions of the query digital image outside the area to emphasize, wherein the first weight is heavier than the second weight;
identifying, from a digital image database and based on the deep neural network-based representation of the query digital image, one or more digital images similar to the query digital image and that emphasize the area of the query digital image; and
providing, in response to the search request, the one or more digital images similar to the query digital image and that emphasize the area of the query digital image.

2. The computer-implemented method of claim 1 , wherein receiving the indication of the area of the query digital image to emphasize comprises receiving an image mask that defines the area of the query digital image to emphasize.

3. The computer-implemented method of claim 1 , wherein generating the deep neural network-based representation comprises:
extracting deep features from the query digital image utilizing a neural network; and
utilizing location-weighted global average pooling to combine the deep features extracted from the query digital image in a manner that weights the deep features corresponding to the area of the query digital image to emphasize with the first weight and that weights the deep features corresponding to the portions of the query digital image outside the area to emphasize with the second weight.

4. The computer-implemented method of claim 3 , further comprising:
receiving a selection of a composition similarity option;
wherein generating the deep neural network-based representation of the query digital image comprises spatially pooling the deep features of query digital image; and
wherein identifying the one or more digital images similar to the query digital image and that emphasize the area of the query digital image based on the deep neural network-based representation of the query digital image comprises identifying digital images having a composition similar to the query digital image.

5. The computer-implemented method of claim 3 , further comprising:
receiving a selection of an objects count option;
wherein generating the deep neural network-based representation of the query digital image comprises utilizing a subitizing neural network to extract features from the query digital image with correspondence to object counts; and
wherein identifying the one or more digital images similar to the query digital image and that emphasize the area of the query digital image comprises identifying digital images having a count of objects that matches a count of objects in the query digital image.

6. The computer-implemented method of claim 1 , further comprising:
receiving a revised search request comprising a second query digital image;
extracting deep features from the second query digital image;
generating a multi-query vector representation that represents a composite of the query digital image and the second query digital image by combining the deep features of the second query digital image and the deep neural network-based representation of the query digital image;
identifying, from the digital image database, one or more additional digital images similar to the query digital image and the second query digital image that emphasize the area of the query digital image based on the multi-query vector representation; and
provide, in response to the revised search request, the one or more additional digital images similar to the query digital image and the second query digital image that emphasize the area of the query digital image.

7. The computer-implemented method of claim 6 , further comprising:
receiving an indication of an area of the second query digital image to emphasize;
generating a deep neural network-based representation of the second query digital image by weighting deep features corresponding to the area of the second query digital image to emphasize utilizing a third weight and weighting deep features corresponding to portions of the second query digital image outside of the area of the second query digital image to emphasize utilizing a fourth weight, wherein the third weight is heavier than the fourth weight;
generating the multi-query vector representation by combining the deep neural network-based representation of the query digital image and the deep neural network-based representation of the second query digital image;
identifying, from the digital image database, at least one digital image similar to the query digital image and the second query digital image that emphasizes the area of the query digital image and the area of the second query digital image; and
provide, in response receiving the indication of the area of the second query digital image, the at least one digital image similar to the query digital image and the second query digital image that emphasizes the area of the query digital image and the area of the second query digital image.

8. A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause a computer device to:
receive a search request comprising a query digital image and an indication of an area of the query digital image to emphasize in a search;
generate a set of deep features for the query digital image utilizing a neural network;
generate a deep neural network-based representation of the query digital image by:
weighting deep features from the set of deep features of the query digital image based corresponding to the area of the query digital image to emphasize utilizing a first weight; and
weighting deep features from the set of deep features of the query digital image corresponding to portions of the query digital image outside the area to emphasize utilizing a second weight, wherein the first weight is heavier than the second weight;

identify, from a digital image database, one or more digital images similar to the query digital image that emphasize the area of the query digital image based on the deep neural network-based representation of the query digital image; and
provide, in response to the search request, the one or more digital images similar to the query digital image that emphasize the area of the query digital image.

9. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the processor, cause the computer device to receive the indication of the area of the query digital image to emphasize in the search by receiving an image mask that defines the area of the query digital image to emphasize.

10. The non-transitory computer readable medium of claim 8 , wherein the instructions, when executed by the processor, cause the computer device to generate the deep neural network-based representation utilizing location-weighted global average pooling to combine the deep features in a manner that weights the area of the query digital image to emphasize with the first weight and that weights the deep features corresponding to the portions of the query digital image outside the area to emphasize with the second weight.

11. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the processor, cause th

Assignees

• Adobe Inc

Inventors

• Lin Zhe
• Shen Xiaohui
• Ling Mingyang
• Zhang Jianming
• Kuen Jason
• Butterfield Brett

Classifications

• G06V20/00 Primary

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/771

  Feature selection, e.g. selecting representative features from a multi-dimensional feature space · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title