Systems and methods for image classification by correlating contextual cues with images

Patent metadata

Field	Value
Publication number	US-10169686-B2
Application number	US-201313959446-A
Country	US
Kind code	B2
Filing date	Aug 5, 2013
Priority date	Aug 5, 2013
Publication date	Jan 1, 2019
Grant date	Jan 1, 2019

Abstract

A sample set of images is received. Each image in the sample set may be associated with one or more social cues. Correlation of each image in the sample set with an image class is scored based on the one or more social cues associated with the image. Based on the scoring, a training set of images to train a classifier is determined from the sample set. In an embodiment, an extent to which an evaluation set of images correlates with the image class is determined. The determination may comprise ranking a top scoring subset of the evaluation set of images.

First claim

What is claimed:

1. A system comprising:
at least one processor; and
a memory storing instructions configured to instruct the at least one processor to perform:
receiving a sample set of images from which images are selected to train an image classifier associated with a plurality of image classes, at least one image in the sample set associated with one or more social cues, the one or more social cues comprising reliability of at least one of a generator or a tagger of the at least one image based on interactions of the at least one of the generator or the tagger on a social networking system;
scoring correlation of the at least one image in the sample set with an image class of the plurality of image classes based on the one or more social cues, wherein
each image class of the plurality of image classes is associated with depiction of an object, action, or concept, and
the scoring comprises generating a score indicative of an extent to which the at least one image depicts the image class; and

determining a training set of images to train the classifier from the sample set based on the scoring.

2. The system of claim 1 , further comprising specifying the image class.

3. The system of claim 1 , wherein the determining comprises ranking each image in the sample set of images based on the scoring.

4. The system of claim 1 , wherein the determining comprises selecting a top scoring subset of the sample set of images.

5. The system of claim 4 , wherein the top scoring subset is the training set of images.

6. The system of claim 1 , further comprising training the classifier based on the training set of images.

7. The system of claim 1 , further comprising generating a visual pattern template associated with the image class.

8. The system of claim 1 , wherein the classifier is configured to use a bag of visual words image classification technique or a neural network image classification technique.

9. The system of claim 1 , further comprising determining an extent to which an evaluation set of images correlates with the image class.

10. The system of claim 9 , wherein the evaluation set of images is different from the sample set of images.

11. The system of claim 9 , wherein the evaluation set of images comprises a larger set of images than the sample set of images.

12. The system of claim 9 , further comprising scoring correlation of each image of the evaluation set of images with a visual pattern template associated with the image class.

13. The system of claim 12 , further comprising ranking each image of the evaluation set based on the scoring correlation of each image of the evaluation set of images.

14. The system of claim 12 , further comprising associating a top scoring subset of the evaluation set of images with the image class.

15. The system of claim 1 , wherein the one or more social cues comprises one or more image tags.

16. The system of claim 15 , further comprising determining a number of instances of a particular image tag among a total number of the one or more image tags associated with an image.

17. The system of claim 1 , wherein the one or more social cues comprises one or more of: location data associated with an image of the sample set of images; or an identity of a tagger or an uploader of the image of the sample set of images.

18. The system of claim 1 , wherein the one or more social cues are received by a social networking system.

19. A computer implemented method comprising:
receiving, by a computer system, a sample set of images from which images are selected to train an image classifier associated with a plurality of image classes, at least one image in the sample set associated with one or more social cues, the one or more social cues comprising reliability of at least one of a generator or a tagger of the at least one image based on interactions of the at least one of the generator or the tagger on a social networking system;
scoring, by the computer system, correlation of the at least one image in the sample set with an image class of the plurality of image classes based on the one or more social cues, wherein
each image class of the plurality of image classes is associated with depiction of an object, action, or concept, and
the scoring comprises generating a score indicative of an extent to which the at least one image depicts the image class; and

determining, by the computer system, a training set of images to train the classifier from the sample set based on the scoring.

20. A non-transitory computer storage medium storing computer-executable instructions that, when executed, cause a computer system to perform a computer-implemented method comprising:
receiving a sample set of images from which images are selected to train an image classifier associated with a plurality of image classes, at least one image in the sample set associated with one or more social cues, the one or more social cues comprising reliability of at least one of a generator or a tagger of the at least one image based on interactions of the at least one of the generator or the tagger on a social networking system;
scoring correlation of the at least one image in the sample set with an image class of the plurality of image classes based on the one or more social cues, wherein
each image class of the plurality of image classes is associated with depiction of an object, action, or concept, and
the scoring comprises generating a score indicative of an extent to which the at least one image depicts the image class; and

determining a training set of images to train the classifier from the sample set based on the scoring.

Assignees

• Facebook Inc

Inventors

• Bourdev Lubomir
• Paluri Balamanohar

Classifications

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F16/24573

  using data annotations, e.g. user-defined metadata · CPC title