User drawing based image search

Patent metadata

Field	Value
Publication number	US-11144587-B2
Application number	US-201615064522-A
Country	US
Kind code	B2
Filing date	Mar 8, 2016
Priority date	Mar 8, 2016
Publication date	Oct 12, 2021
Grant date	Oct 12, 2021

Abstract

Various aspects of the subject technology relate to systems, methods, and machine-readable media for user drawing based image search. These aspects include an image retrieval system using a convolutional neural network trained to identify how users draw semantic concepts and using an image search engine to search against images having a similar concept. The aspects include mapping between concepts of the user drawing space and concepts of the image space such that images associated with the same concept are identified. For each input user drawing, the drawing is first processed through a concept classifier to identify a corresponding concept, and then through a feature extractor to form a corresponding feature vector. The results from the concept classifier and the feature extractor may be combined to search against a collection of images having a similar concept to determine a listing of images ranked by visual and semantic similarity to the input.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
generating a library of training drawings;
extracting visual feature descriptors of each training drawing in the library of training drawings;
mapping the visual feature descriptors of each training drawing to a training vector;
associating a concept of a plurality of predetermined concepts to each training vector;
receiving a first user input identifying a representation of a user drawing, the user drawing representing a search query for initiating an image search;
determining a concept of the user drawing, the concept of the user drawing indicating a semantic subject matter in the representation of the user drawing;
determining a collection of images is relevant to the concept of the user drawing;
determining a subset of the collection of images associated with the concept of the user drawing;
mapping the concept of the user drawing to a cluster of training drawings having a cluster identifier that matches the semantic subject matter,
wherein the cluster identifier is a semantic concept corresponding to the cluster of training drawings,
wherein the subset of the collection of images includes an image data from the cluster of training drawings, and
wherein the image data is indicative of how users draw semantic concepts;
comparing, in response to the determination that the subset is associated with the concept of the user drawing, feature vectors of images in the subset of the collection of images to a feature vector of the user drawing;
storing, in a memory associated with the library of training drawings and prior to receiving a second user input of the user drawing, the visual feature descriptor of the representation of the user drawing for offline training in determining a subsequent concept of the user drawing that combines the first user input and the second user input,
wherein the offline training is performed prior to receiving the second user input, and
wherein the second user input is an extension of the user drawing and is received subsequently to the first user input; and
generating search results associated with the image search based on comparison results of the comparison between the feature vector of the user drawing and a feature vector of an image in the subset of the collection of images.

2. The computer-implemented method of claim 1 , wherein receiving the first user input comprises:
detecting a first sequence of user inputs identifying a first portion of the user drawing, the first sequence of user inputs comprising a plurality of pixel coordinates corresponding to a two-dimensional representation of a first drawing stroke.

3. The computer-implemented method of claim 2 , wherein the concept is determined in response to the detection of the first sequence of user inputs.

4. The computer-implemented method of claim 2 , further comprising:
detecting a second sequence of user inputs identifying a second portion of the user drawing, the second sequence of user inputs being detected subsequent to the first sequence of user inputs, the second sequence of user inputs comprising a second plurality of pixel coordinates corresponding to a two-dimensional representation of a second drawing stroke.

5. The computer-implemented method of claim 4 , wherein the concept determined based on the first sequence of user inputs is changed to a new concept in response to the detection of the second sequence of user inputs.

6. The computer-implemented method of claim 4 , wherein the concept is determined based on a combination of the first sequence of user inputs and the second sequence of user inputs.

7. The computer-implemented method of claim 1 , wherein determining the concept of the user drawing comprises:
comparing the representation of the user drawing to a plurality of training drawings, each of the plurality of training drawings being associated with a concept of the plurality of predetermined concepts.

8. The computer-implemented method of claim 1 , wherein comparing the feature vectors of the images in the subset of the collection of images to the feature vector of the user drawing comprises:
determining a vector distance between the feature vector of the user drawing and each of the feature vectors of the images; and
selecting a feature vector of an image that corresponds to a smallest vector distance.

9. The computer-implemented method of claim 8 , wherein generating the search results comprises:
obtaining images from the collection of images which correspond to a feature vector of an image in the subset of the collection of images; and
providing the images for display to a user of a client device.

10. A system comprising:
one or more processors;
a non-transitory computer-readable storage medium coupled to the one or more processors, the non-transitory computer-readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to:
generate a library of training drawings;
map a concept of each training drawing to a training vector;
associate a concept of a plurality of predetermined concepts to each training vector;
detect a first input query identifying a representation of a first user drawing;
initiate, in response to detecting the first input query, a first search for images relevant to a concept associated with the first user drawing, wherein the first search includes:
(i) determining a subset of a collection of images is associated with the concept associated with the first user drawing,
(ii) mapping the concept associated with the first user drawing to a cluster of training drawings having a cluster identifier that matches a concept associated with a first user drawing,
wherein the cluster identifier is a semantic concept corresponding to the cluster of training drawings,
wherein the subset of the collection of images includes an image data from the cluster of training drawings, and
wherein the image data is indicative of how users draw semantic concepts,
(iii) comparing, in response to the determination that the subset is associated with the concept associated with the first user drawing, a feature vector of multiple images in the subset of the collection of images to a feature vector of the first user drawing, and
(iv) storing, in a memory associated with the library of training drawings and prior to receiving a second input query of the first user drawing, the concept associated with the first user drawing for offline training in determining a subsequent concept associated with the first user drawing that combines the first input query and the second input query,
wherein the offline training is performed prior to receiving the second input query, and
wherein the second input query is received subsequently to the first input query;

generate first search results associated with the first search;
provide, for display, the first search results;
detect the second input query identifying a representation of the first user drawing as a combination of the first input query and the second input query, the second input query being detected subsequent to the first input query;
initiate, in response to detecting the second input query, a second search for images relevant to the subsequent concept associated with the first user drawing as the combination of the first input query and the second input query;
generate second search results associated with the second search; and
provide, for display, the second search results, the second search results being different than the first search results if a subsequent concept associated with a second user drawing as the combination of the first input query and the second input que

Assignees

• Shutterstock Inc

Inventors

• Chester David
• Hurst Nathan

Classifications

• G06F16/56

  having vectorial format · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title

• G06F16/532 Primary

  Query formulation, e.g. graphical querying · CPC title

• G06F16/5838

  using colour · CPC title

• G06F16/54

  Browsing; Visualisation therefor · CPC title