Fine-grained categorization

Patent metadata

Field	Value
Publication number	US-9818048-B2
Application number	US-201514967876-A
Country	US
Kind code	B2
Filing date	Dec 14, 2015
Priority date	Jan 19, 2015
Publication date	Nov 14, 2017
Grant date	Nov 14, 2017

Abstract

An image is passed through an image identifier to identify a coarse category for the image and a bounding box for a categorized object. A mask is used to identify the portion of the image that represents the object. Given the foreground mask, the convex hull of the mask is located and an aligned rectangle of minimum area that encloses the hull is fitted. The aligned bounding box is rotated and scaled, so that the foreground object is roughly moved to a standard orientation and size (referred to as calibrated). The calibrated image is used as an input to a fine-grained categorization module, which determines the fine category within the coarse category for the input image.

First claim

What is claimed is:

1. A system comprising:
a memory having instructions embodied thereon; and
one or more processors configured by the instructions to perform operations comprising:
identifying a coarse category based on a first image;
generating a bounding box that encompasses an item depicted in the first image;
generating a second image by cropping the first image to the bounding box and rotating the cropped portion; and
identifying a fine category based on the second image, the fine category being a sub-category of the coarse category.

2. The system of claim 1 , wherein the operations further comprise:
receiving the first image;
selecting an advertisement based on the fine category; and
causing display of the first image and the advertisement.

3. The system of claim 2 , wherein:
the first image is associated with a first user account; and
the display of the first image and the advertisement is to a second user account.

4. The system of claim 1 , wherein the generating of the bounding box that encompasses the item depicted in the first image comprises:
generating a mask for the item depicted in the first image;
generating a convex hull for the mask; and
generating a bounding box that encompasses the convex hull.

5. The system of claim 1 , wherein:
the fine category is selected from a set of available fine categories for the coarse category;
the operations further comprise training a set of fragment detectors for the coarse category using operations comprising:
accessing a training set of images for each of the available fine categories;
accessing a validation set of images for each of the available fine categories; and
identifying, from the training set of images and the validation set of images, a discriminative fragment set for each of the available fine categories; and
the identifying of the fine category based on the second image comprises providing the second image to the trained set of fragment detectors for the coarse category.

6. The system of claim 5 , wherein the identifying of the discriminative fragment sets for each of the available fine categories for the coarse category comprises:
for each of the available fine categories for the coarse category:
extracting fragments from the training set of images for the fine category; and
extracting fragments from the validation set of images for the fine category.

7. The system of claim 6 , wherein the identifying of the discriminative fragment sets for each of the available fine categories further comprises:
for each of the available fine categories:
selecting a random subset of the fragments extracted from the training set of images for the fine category; and
selecting a random subset of the fragments extracted from the validation set of mages for the fine category.

8. The system of claim 1 , wherein the operations further comprise:
receiving the first image from a client device;
identifying a set of images based on the fine category; and
causing the set of images to be displayed on the client device.

9. The system of claim 8 , wherein:
the identifying of the set of images based on the fine category comprises identifying images associated with the fine category.

10. The system of claim 8 , wherein:
the identifying of the set of images based on the fine category comprises identifying images of items for sale in an online marketplace.

11. A method comprising:
identifying a coarse category based on a first image;
generating a bounding box that encompasses an item depicted in the first image;
generating a second image by cropping the first image to the bounding box and rotating the cropped portion; and
identifying, by a processor of a machine, a fine category based on the second image, the fine category being a sub-category of the coarse category.

12. The method of claim 11 , wherein the method further comprises:
receiving the first image;
selecting an advertisement based on the fine category; and
causing display of the first image and the advertisement.

13. The method of claim 12 , wherein:
the first image is associated with a first user account; and
the display of the first image and the advertisement is to a second user account.

14. The method of claim 11 , wherein the generating of the bounding box that encompasses the item depicted in the first image comprises:
generating a mask for the item depicted in the first image;
generating a convex hull for the mask; and
generating a bounding box that encompasses the convex hull.

15. The method of claim 11 , wherein:
the fine category is selected from a set of available fine categories for the coarse category;
the method further comprises training a set of fragment detectors for the coarse category using operations comprising:
accessing a training set of images for each of the available fine categories;
accessing a validation set of images for each of the available fine categories; and
identifying, from the training set of images and the validation set of images, discriminative fragment sets for each of the available fine categories; and
the identifying of the fine category of the item in the new image comprises providing the second image to the trained set of fragment detectors for the coarse category.

16. The method of claim 15 , wherein the identifying of the discriminative fragment sets for each of the available fine categories for the coarse category comprises:
for each of the available fine categories for the coarse category:
extracting fragments from the training set of images for the fine category; and
extracting fragments from the validation set of images for the fine category.

17. The method of claim 16 , wherein the identifying of the discriminative fragment sets for each of the available fine categories further comprises:
for each of the available fine categories:
selecting a random subset of the fragments extracted from the training set of images for the fine category; and
selecting a random subset of the fragments extracted from the validation set of images for the fine category.

18. The method of claim 11 , further comprising:
receiving the image from a client device;
identifying a set of images based on the fine category; and
causing the set of images to be displayed on the client device.

19. The method of claim 18 , wherein:
the identifying of the set of images based on the fine category comprises identifying images associated with the fine category.

20. A machine-readable hardware storage device having instructions embodied thereon, that when executed by a processor of a machine, cause the machine to perform operations comprising:
identifying a coarse category based on a first image;
generating a bounding box that encompasses an item depicted in the first image;
generating a second image by cropping the first image to the bounding box and rotating the cropped portion; and
identifying a fine category based on the second image, the fine category being a sub-category of the coarse category.

Assignees

• Ebay Inc

Inventors

• Kiapour M Hadi
• Di Wei
• Jagadeesh Vignesh
• Piramuthu Robinson

Classifications

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/24147 Primary

  Distances to closest patterns, e.g. nearest neighbour classification · CPC title

• G06V10/245 Primary

  by locating a pattern; Special marks for positioning · CPC title

• G06F18/2111

  by using evolutionary computational techniques, e.g. genetic algorithms · CPC title

• G06F18/2411 Primary

  based on the proximity to a decision surface, e.g. support vector machines · CPC title