Hand-drawn sketch recognition

Patent metadata

Field	Value
Publication number	US-9870516-B2
Application number	US-201514847834-A
Country	US
Kind code	B2
Filing date	Sep 8, 2015
Priority date	May 3, 2013
Publication date	Jan 16, 2018
Grant date	Jan 16, 2018

Abstract

Some examples of a sketch-based image recognition system may generate a model for identifying a subject of a sketch. The model is formed from a plurality of images having visual features similar to the visual features of the sketch. The model may include object topics representative of categories which may correspond to the subject of the sketch and shape topics representative of the visual features of the sketch.

First claim

The invention claimed is:

1. A method comprising:
receiving, from a computing device over a network, a sketch;
searching a database of images using the sketch as a query to identify a set of images, an individual image of the set of images including a shape feature similar to at least one shape feature of the sketch;
determining, based at least in part on the set of images, at least one object topic; and
generating, based at least in part on the set of images, a model for identifying a subject of the sketch, the model including:
the at least one object topic, the at least one object topic being representative of a category that is descriptive of the subject of the sketch; and
at least one shape topic associated with the at least one object topic, the at least one shape topic representative of a shape feature of the sketch.

2. The method of claim 1 , wherein the individual image of the set of images further includes at least one text-based label, and wherein the method further comprises:
generating a word distribution for the at least one object topic from text-based labels of at least a portion of the set of images; and
identifying a word descriptive of the subject of the sketch from the word distribution.

3. The method of claim 2 , wherein the word distribution includes at least a first word and a second word, and wherein the method further comprises:
determining a first rank for the first word based at least in part on a first probability that the first word is descriptive of the subject of the sketch;
determining a second rank for the second word based at least in part on a second probability that the second word is descriptive of the subject of the sketch; and
generating a ranking for the first word and the second word based at least in part on the first rank and the second rank,
and wherein identifying the word descriptive of the subject of the sketch is based at least in part on the ranking.

4. The method of claim 1 , wherein the at least one object topic includes a first object topic and a second object topic, and wherein the method further comprises:
determining a first probability that a shape topic of the at least one shape topic belongs to the first object topic;
determining a second probability that the shape topic belongs to the second object topic; and
associating the shape topic with at least one of the first object topic or the second object topic based at least in part on the first probability and the second probability.

5. The method of claim 4 , wherein:
the first probability is greater than the second probability; and
associating the shape topic with the at least one of the first object topic or the second object topic comprises associating the shape topic with the first object topic based at least in part on the first probability for the first shape topic being greater than the second probability for the second object topic.

6. The method of claim 1 , further comprising:
filtering the set of images into a set of filtered images by removing images that include more than one subject,
and wherein generating the model based at least in part on the set of images comprises generating the model based at least in part on the set of filtered images.

7. The method of claim 1 , further comprising:
identifying, for at least an image of the set of images, a shape feature of the image; and
associating the image with a shape topic of the at least one shape topic based at least in part on the shape feature of the image.

8. A computing device comprising:
one or more input interfaces for receiving a sketch;
one or more processors; and
a computer-readable storage media storing instructions, which when executed by the one or more processors, cause the one or more processors to perform operations comprising:
performing an image-based search of a database of images using the sketch as a search query;
identifying a set of images, an individual image of the set of images including a shape feature similar to a shape feature of the sketch;
determining an object topic based at least in part on the set of images; and
generating, based at least in part on the set of images, a model for identifying a subject associated with the sketch, the model including:
the object topic, the object topic representative of a category for the subject that is associated with the sketch; and
a shape topic associated with the object topic, the shape topic representative of the shape feature of the sketch.

9. The computing device of claim 8 , wherein the individual image of the set of images is associated with at least one text-based label, and wherein the operations further comprise:
generating a word distribution for the object topic from text-based labels associated with at least a portion of the set of images; and
identifying a word descriptive of the subject associated with the sketch from the word distribution.

10. The computing device of claim 9 , wherein the word distribution includes at least a first word and a second word, and wherein the operations further comprise:
determining a first rank for the first word based at least in part on a first probability that the first word is descriptive of the subject associated with the sketch;
determining a second rank for the second word based at least in part on a second probability that the second word is descriptive of the subject associated with the sketch; and
generating a ranking for the first word and the second word based at least in part on the first rank and the second rank,
and wherein identifying the word descriptive of the subject of the sketch is based at least in part on the ranking.

11. The computing device of claim 8 , wherein the model further includes an additional object topic, and wherein the operations further comprise:
determining a first probability that the shape topic is associated with the object topic;
determining a second probability that the shape topic is associated with the additional object topic; and
associating the shape topic with the object topic based at least in part on the first probability and the second probability.

12. The computing device of claim 8 , the operations further comprising:
filtering the set of images into a set of filtered images by removing images that include more than one subject,
and wherein generating the model based at least in part on the set of images comprises generating the model based at least in part on the set of filtered images.

13. The computing device of claim 8 , the operations further comprising:
identifying, for at least an image of the set of images, a shape feature of the image; and
associating the image with the shape topic based at least in part on the shape feature of the image.

14. A computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
searching a database of images using a sketch as a query to identify a set of images, an individual image of the set of images including a shape feature similar to at least one shape feature of the sketch;
determining, based at least in part on the set of images, at least one object topic; and
generating, based at least in part on the set of images, a model for identifying a subject of the sketch, the model including:
the at least one object topic, the at least one object topic being representative of a category that is descriptive of the subject of the sketch; and
at least one shape topic associated with the at least one object topic, the at least one shape topic representative of a shape feature of the sketch.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Wang Changhu
• Zhang Lei
• Sun Zhenbang

Classifications

• G06F16/5854 Primary

  using shape and object relationship · CPC title

• G06K9/6202 Primary

  Physics · mapped topic

• G06F17/30259

  Physics · mapped topic