Fine-grained categorization

Patent metadata

Field	Value
Publication number	US-2018039866-A1
Application number	US-201715788115-A
Country	US
Kind code	A1
Filing date	Oct 19, 2017
Priority date	Jan 19, 2015
Publication date	Feb 8, 2018
Grant date	—

Abstract

An image is passed through an image identifier to identify a coarse category for the image and a bounding box for a categorized object. A mask is used to identify the portion of the image that represents the object. Given the foreground mask, the convex hull of the mask is located and an aligned rectangle of minimum area that encloses the hull is fitted. The aligned bounding box is rotated and scaled, so that the foreground object is roughly moved to a standard orientation and size (referred to as calibrated). The calibrated image is used as an input to a fine-grained categorization module, which determines the fine category within the coarse category for the input image.

First claim

1 . A system comprising:
a memory having instructions embodied thereon; and one or more processors configured by the instructions to perform operations comprising:
receiving an image;
identifying a coarse category of an item in the image;
based on the identified coarse category, analyzing the image to identify a fine category within the coarse category, the fine category being a sub-category of the coarse category; and
causing presentation of information related to the identified fine category for the item.

2 . The system of claim 1 , wherein the operations further comprise accessing one or more attributes associated with the fine category, the one or more attributes being provided by a user and being used to supplement or replace fine category information used to perform a search.

3 . The system of claim 1 , wherein the operations further comprise performing a search for the information related to the identified fine category for the item.

4 . The system of claim 1 , wherein the operations further comprise:
generating a bounding box that encompasses the item in the image; and generating a second image by cropping the image to the bounding box and rotating the cropped portion.

5 . The system of claim 4 , wherein the generating of the bounding box that encompasses the item comprises:
generating a mask for the item in the image; and generating a convex hull for the mask, wherein the bounding box encompasses the convex hull.

6 . The system of claim 1 , wherein:
the fine category is selected from a set of available fine categories for the coarse category; the operations further comprise training a set of fragment detectors for the coarse category using operations comprising:
accessing a training set of images for each of the available fine categories;
accessing a validation set of images for each of the available fine categories; and
identifying, from the training set of images and the validation set of images, a discriminative fragment set for each of the available fine categories; and
the analyzing the image to identify the fine category comprises providing the second image to the trained set of fragment detectors for the coarse category.

7 . The system of claim 6 , wherein the identifying of the discriminative fragment set for each of the available fine categories comprises:
for each of the available fine categories: extracting fragments from the training set of images for the fine category; and extracting fragments from the validation set of images for the fine category.

8 . The system of claim 6 , wherein the identifying of the discriminative fragment sets for each of the available fine categories comprises:
for each of the available fine categories: selecting a random subset of the fragments extracted from the training set of images for the fine category; and selecting a random subset of the fragments extracted from the validation set of images for the fine category.

9 . A method comprising:
receiving an image; identifying, using one or more hardware processors, a coarse category of an item in the image; based on the identified coarse category, analyzing the image to identify a fine category within the coarse category, the fine category being a sub-category of the coarse category; and causing presentation of information related to the identified fine category.

10 . The method of claim 9 , further comprising accessing one or more attributes associated with the fine category, the one or more attributes being provided by a user and being used to supplement or replace fine category information used to perform a search.

11 . The method of claim 9 , further comprising performing a search for the information related to the identified fine category for the item.

12 . The method of claim 9 , further comprising:
generating a bounding box that encompasses the item in the image; and generating a second image by cropping the image to the bounding box and rotating the cropped portion.

13 . The method of claim 12 , wherein the generating of the bounding box that encompasses the item comprises:
generating a mask for the item in the image; and generating a convex hull for the mask, wherein the bounding box encompasses the convex hull.

14 . The method of claim 9 , wherein:
the fine category is selected from a set of available fine categories for the coarse category; the method further comprising training a set of fragment detectors for the coarse category using operations comprising:
accessing a training set of images for each of the available fine categories;
accessing a validation set of images for each of the available fine categories; and
identifying, from the training set of images and the validation set of images, a discriminative fragment set for each of the available fine categories; and
the analyzing the image to identify the fine category comprises providing the second image to the trained set of fragment detectors for the coarse category.

15 . The method of claim 14 , wherein the identifying of the discriminative fragment set for each of the available fine categories comprises:
for each of the available fine categories: extracting fragments from the training set of images for the fine category; and extracting fragments from the validation set of images for the fine category.

16 . The method of claim 14 , wherein the identifying of the discriminative fragment sets for each of the available fine categories comprises:
for each of the available fine categories: selecting a random subset of the fragments extracted from the training set of images for the fine category; and selecting a random subset of the fragments extracted from the validation set of images for the fine category.

17 . A machine-readable storage device storing instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving an image; identifying a coarse category of an item in the image; based on the identified coarse category, analyzing the image to identify a fine category within the coarse category, the fine category being a sub-category of the coarse category; and causing presentation of information related to the identified fine category.

18 . The machine-readable storage device of claim 17 , wherein the operations further comprise:
generating a bounding box that encompasses the item in the image; and generating a second image by cropping the image to the bounding box and rotating the cropped portion.

19 . The machine-readable storage device of claim 18 , wherein the generating of the bounding box that encompasses the item comprises:
generating a mask for the item in the image; and generating a convex hull for the mask, wherein the bounding box encompasses the convex hull.

20 . The machine-readable storage device of claim 17 , wherein:
the fine category is selected from a set of available fine categories for the coarse category; the operations further comprising training a set of fragment detectors for the coarse category using operations comprising:
accessing a training set of images for each of the available fine categories;
accessing a validation set of images for each of the available fine categories; and
identifying, from the training set of images and the validation set of images, a discriminative fragment set for each of the available fine categories; and
the analyzing the image to identify the fine category comprises providing the second image to the traine

Assignees

• Ebay Inc

Inventors

• Kiapour Mohammadhadi
• Di Wei
• Jagadeesh Vignesh
• Piramuthu Robinson

Classifications

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/24147 Primary

  Distances to closest patterns, e.g. nearest neighbour classification · CPC title

• G06V10/245 Primary

  by locating a pattern; Special marks for positioning · CPC title

• G06F18/2411 Primary

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06F18/2111

  by using evolutionary computational techniques, e.g. genetic algorithms · CPC title