Identification of a salient portion of an image

Patent metadata

Field	Value
Publication number	US-10437878-B2
Application number	US-201615393206-A
Country	US
Kind code	B2
Filing date	Dec 28, 2016
Priority date	Dec 28, 2016
Publication date	Oct 8, 2019
Grant date	Oct 8, 2019

Abstract

Various aspects of the subject technology relate to systems, methods, and machine-readable media for identification of a salient portion of an image. A system may receive user input identifying a search query for content from a client device. The system may determine a listing of images responsive to the search query from an image collection. The system may obtain one or more image crops for at least one image of the listing of images based on a saliency map of the at least one image. In one or more implementations, each of the one or more image crops indicates a salient region of a corresponding image. The system may provide a set of search results responsive to the search query to the client device. In one or more implementations, the set of search results includes the obtained one or more image crops in a prioritized listing of image crops.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving user input identifying a search query for content from a client device;
determining a listing of images responsive to the search query from an image collection;
providing at least one image of the listing of images to a trained convolutional neural network that determines a saliency of each pixel in the at least one image;
obtaining one or more image crops for the at least one based on a saliency map of the at least one image, each of the one or more image crops indicating a salient region of a corresponding image;
determining an integral image of the at least one image, each pixel value of the integral image being different from that of the at least one image based on the saliency map;
for each fixed region with a predetermined aspect ratio within the at least one image, determining a sum of saliency for the fixed region based on the integral image;
modifying the at least one image by cropping a region of the at least one image that corresponds to the fixed region having a maximum sum of saliency, the region of the at least one image being cropped to one of a plurality of predetermined aspect ratios; and
providing a set of search results responsive to the search query to the client device, the set of search results including the obtained one or more image crops in a prioritized listing of image crops.

2. The computer-implemented method of claim 1 , further comprising:
determining the saliency of each pixel in the at least one image with respect to an overall scene of the at least one image; and
generating the saliency map of the at least one image using the trained convolutional neural network.

3. The computer-implemented method of claim 1 , further comprising:
providing each image from the image collection to a saliency model;
identifying salient portions of the image using the saliency model;
determining one or more image crops for the image from the identified salient portions of the image; and
providing one or more tuples of image information for each image, each of the one or more tuples of image information indicating a respective image crop of the determined one or more image crops.

4. The computer-implemented method of claim 3 , wherein each of the one or more tuples of image information include one or more of an image identifier for the image, an identifier indicating one image crop from the one or more image crops, and coordinates of the image crop.

5. The computer-implemented method of claim 3 , further comprising:
providing metadata from each image in the image collection to a saliency index data structure;
providing each of the one or more tuples of image information to the saliency index data structure;
generating an association between each of the one or more tuples of image information with a corresponding image of the image collection in the saliency index data structure; and
providing an index for each of generated association in the saliency index data structure.

6. The computer-implemented method of claim 5 , further comprising:
providing the search query to an image search engine, the search query indicating a search term;
searching the search term against index entries of the saliency index data structure;
determining that the search term corresponds to the metadata from at least one index entry of the saliency index data structure;
determining a score for each image crop associated with an image from the at least one index entry using the metadata associated with the image; and
providing a listing of images and a listing of image crops responsive to the search query, the listing of image crops being prioritized within the listing of images based on the determined scores, each image crop in the listing of image crops indicating a salient region of a corresponding image of the image collection.

7. The computer-implemented method of claim 6 , wherein the determined score includes a first score of the image, a second score of the image crop and a weighting for the second score of the image crop.

8. The computer-implemented method of claim 5 , further comprising:
providing the search query to an image search engine;
obtaining the listing of images based on the provided search query, at least one image from the listing of images having an aspect ratio that exceeds a predetermined aspect ratio threshold;
determining whether one or more image crops associated with the at least one of the listing of images are indexed;
obtaining at least one of the one or more image crops from the saliency index data structure when it is determined that the one or more image crops associated with the at least one image are indexed;
generating the one or more image crops from the at least one image based on the saliency map of the at least one image; and
providing the one or more image crops in a prioritized listing of image crops, each of the one or more image crops having an aspect ratio that does not exceed the predetermined aspect ratio threshold, each of the one or more image crops indicating a salient region of a corresponding of the image collection.

9. The computer-implemented method of claim 1 , further comprising:
providing the at least one image from the listing of images to a trained convolutional neural network that determines a saliency of each pixel in the at least one image with respect to an overall scene of the at least one image;
generating the saliency map of the at least one image using the trained convolutional neural network;
for each pixel in the at least one image, modifying a value of the pixel using a weighted filter to generate a masking for a non-salient region of the at least one image based on the saliency map; and
providing the at least one image with the masking over the non-salient region of the at least one image.

10. The computer-implemented method of claim 1 , further comprising:
providing a plurality of sets of training images to a computer-operated convolutional neural network, wherein the computer-operated convolutional neural network processes the plurality of sets of training images to learn to identify features relating to at least one of a plurality of object classes, wherein each of the plurality of sets of training images is associated with one object class of the plurality of object classes;
generating feature vectors for each training image in the plurality of sets of training images using the computer-operated convolutional neural network; and
clustering the feature vectors into a plurality of clusters,
wherein at least one of the plurality of clusters is associated with the search query.

11. The computer-implemented method of claim 10 , further comprising:
generating processed pixel data including the feature vectors from the plurality of sets of training images;
determining a probability using the computer-operated convolutional neural network for an object class, the probability indicating a likelihood that a subject image corresponds to the object class; and
providing an aggregate of probabilities that includes a probability for each object class in a set of object classes.

12. The computer-implemented method of claim 1 , further comprising:
generating feature vectors for each image in the listing of image using a computer-operated convolutional neural network;
generating processed pixel data including the feature vectors for each image from the listing of images;
determining a saliency value using the computer-operated convolutional neural network for each pixel in the image, the saliency value indicating a likelihood that the pixel within the image is salient; and
generating the saliency

Assignees

• Shutterstock Inc

Inventors

• Ranzinger Mike
• Hohwald Heath
• Hurst Nathan

Classifications

• G06V20/188

  Vegetation · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title