Utilizing a deep neural network-based model to identify visually similar digital images based on user-selected visual attributes

Patent metadata

Field	Value
Publication number	US-10628708-B2
Application number	US-201815983949-A
Country	US
Kind code	B2
Filing date	May 18, 2018
Priority date	May 18, 2018
Publication date	Apr 21, 2020
Grant date	Apr 21, 2020

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for utilizing a deep neural network-based model to identify similar digital images for query digital images. For example, the disclosed systems utilize a deep neural network-based model to analyze query digital images to generate deep neural network-based representations of the query digital images. In addition, the disclosed systems can generate results of visually-similar digital images for the query digital images based on comparing the deep neural network-based representations with representations of candidate digital images. Furthermore, the disclosed systems can identify visually similar digital images based on user-defined attributes and image masks to emphasize specific attributes or portions of query digital images.

First claim

What is claimed is:

1. In a digital medium environment for searching digital images, a non-transitory computer readable medium for matching digital images based on visual similarity comprising instructions that, when executed by a processor, cause a computer device to:
receive, from a user client device, a user selection of a query digital image and at least one of spatial selectivity, image composition, or object count to use to identify similar digital images;
receive, from the user client device, an image mask that indicates a portion of the query digital image to emphasize in identifying similar digital images and an image mask weight corresponding to the image mask;
utilize a deep neural network-based model to generate a set of deep features for the query digital image;
generate, based on weighting the set of deep features of the query digital image utilizing the image mask and the image mask weight and in accordance with the user selection, a deep neural network-based representation of the query digital image by utilizing one or more of a spatial selectivity algorithm, an image composition algorithm, or an object count algorithm; and
based on the deep neural network-based representation of the query digital image, identify, from a digital image database, a similar digital image for the query digital image.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the processor, cause the computer device to train the deep neural network-based model to generate deep features for digital images.

3. The non-transitory computer readable medium of claim 1 , wherein the instructions, when executed by the processor, cause the computer device to generate the deep neural network-based representation by modifying the set of deep features by utilizing a deep neural network-based representation comprising at least one of the spatial selectivity algorithm, the image composition algorithm, or the object count algorithm.

4. The non-transitory computer readable medium of claim 3 , further comprising instructions that, when executed by the processor, cause the computer device to receive, from the user client device:
an additional query digital image and an additional image mask that indicates a portion of the additional query digital image; and
an indication to invert the additional image mask to emphasize portions of the additional query digital image that are outside the additional image mask in identifying similar digital images.

5. The non-transitory computer readable medium of claim 4 , further comprising instructions that, when executed by the processor, cause the computer device to generate, based on receiving the additional query digital image, the additional image mask, and the indication to invert the additional image mask, an additional deep neural network-based representation for the additional query digital image based on weighting the portion of the additional query digital image outside of the additional image mask.

6. The non-transitory computer readable medium of claim 5 , wherein weighting the portion of the additional query digital image outside of the additional image mask comprises applying weights to features of a set of features that correspond to the portion of the additional query digital image outside of the additional image mask.

7. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the processor, cause the computer device to:
determine similarity scores for a plurality of digital images from the digital image database; and
rank the plurality of digital images based on the determined similarity scores.

8. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the processor, cause the computer device to:
receive, from the user client device, a selection of a second query digital image;
utilize the deep neural network-based model to generate a second set of deep features for the second query digital image;
generate, based on the second set of deep features of the second query digital image, a multi-query vector representation that represents a composite of the query digital image and the second query digital image; and
identify a similar digital image for the multi-query vector representation from within the digital image database.

9. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the processor, cause the computer device to:
receive, in relation to the second query digital image, a second user selection of at least one of spatial selection, image composition, or object count; and
wherein the instructions cause the computer device to generate, based on the second user selection, the multi-query vector representation by utilizing at least one of the spatial selectivity algorithm, the image composition algorithm, or the object count algorithm.

10. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the processor, cause the computer device to receive, from the user client device:
a second image mask that indicates a portion of the second query digital image to emphasize in identifying similar digital images.

11. The non-transitory computer readable medium of claim 10 , further comprising instructions that, when executed by the processor, cause the computer device to modify, in response to receiving the second image mask, the multi-query vector representation to emphasize the portion of the second query digital image indicated by the second image mask.

12. In a digital medium environment for searching for digital images, a system for matching digital images based on visual similarity comprising:
a processor; and
a non-transitory computer readable medium comprising instructions that, when executed by the processor, cause the system to:
receive, from a user client device, a user selection of a query digital image and at least one of spatial selectivity, image composition, or object count to use to identify similar digital images;
receive, from the user client device, an image mask that indicates a portion of the query digital image to emphasize in identifying similar digital images and an image mask weight corresponding to the image mask;
utilize a deep neural network-based model to generate a set of deep features for the query digital image;
generate, based on weighting the set of deep features of the query digital image utilizing the image mask and the image mask weight and in accordance with the user selection, a deep neural network-based representation of the query digital image by utilizing one or more of a spatial selectivity algorithm, an image composition algorithm, or an object count algorithm;
determine, based on the deep neural network-based representation of the query digital image, similarity scores for a plurality of digital images within a digital image database; and
identify, based on the deep neural network-based representation of the query digital image and further based on the similarity scores of the plurality of digital images, a similar digital image for the query digital image from the digital image database.

13. The system of claim 12 , wherein the instructions, when executed by the processor, cause the system to determine the similarity scores by comparing deep features of each of the plurality of digital images with deep features of the query digital image.

14. The system of claim 13 , further comprising instructions that, when executed by the processor, cause the system to:
receive, fr

Assignees

• Adobe Inc

Inventors

• Lin Zhe
• Shen Xiaohui
• Ling Mingyang
• Zhang Jianming
• Kuen Jason
• Butterfield Brett

Classifications

• G06F16/532 Primary

  Query formulation, e.g. graphical querying · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/75

  involving models · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06F16/5854

  using shape and object relationship · CPC title