Hierarchical deep convolutional neural network for image classification

Patent metadata

Field	Value
Publication number	US-10387773-B2
Application number	US-201414582059-A
Country	US
Kind code	B2
Filing date	Dec 23, 2014
Priority date	Oct 27, 2014
Publication date	Aug 20, 2019
Grant date	Aug 20, 2019

Abstract

Hierarchical branching deep convolutional neural networks (HD-CNNs) improve existing convolutional neural network (CNN) technology. In a HD-CNN, classes that can be easily distinguished are classified in a higher layer coarse category CNN, while the most difficult classifications are done on lower layer fine category CNNs. Multinomial logistic loss and a novel temporal sparsity penalty may be used in HD-CNN training. The use of multinomial logistic loss and a temporal sparsity penalty causes each branching component to deal with distinct subsets of categories.

First claim

What is claimed is:

1. A system comprising:
a memory that stores instructions; and
one or more processors configured by the instructions to perform operations comprising:
accessing a dataset comprising categorized data having a plurality of fine categories;
dividing the dataset into a training set and a value set;
training a first convolutional neural network (CNN) model using the training set;
generating a confusion matrix of the first CNN model using the value set;
applying a clustering algorithm to the confusion matrix to determine, for each fine category, an associated coarse category of a plurality of coarse categories, each coarse category being associated with more than one fine category;
training a base CNN to distinguish between the coarse categories;
training a single fine CNN for each coarse category, the single fine CNN for said coarse category to distinguish between the fine categories associated with said coarse category;
receiving a request to classify data;
using the base CNN, determining a coarse category of the data;
using the single fine CNN for the determined coarse category, determining a single fine category of the data; and
responsive to the request, transmitting the single fine category of the data.

2. The system of claim 1 , wherein the clustering algorithm comprises an affinity propagation algorithm.

3. The system of claim 2 , wherein the operations further comprise:
obtaining low-dimensional feature representations for the fine categories using a Laplacian eigenmap.

4. The system of claim 2 , wherein the training of the fine CNN for each coarse category comprises:
training a second CNN model using the training set;
generating the fine CNN for each coarse category from the second CNN; and
training the fine CNN for each coarse category using a subset of the training set, the subset excluding data having fine categories not associated with said coarse category.

5. The system of claim 1 , wherein the operations further comprise:
combining the base CNN with each of the fine CNNs to form a hierarchically deep CNN (HD-CNN); and
fine-tuning the HD-CNN.

6. The system of claim 5 , wherein the fine-tuning of the HD-CNN comprises:
beginning the fine-tuning with a learning factor;
training the HD-CNN by iterating over a series of training batches using the learning factor;
after each iteration, comparing a training error for the training batch to a threshold;
determining, based on the comparison, that the training error for the training batch is below the threshold; and
in response to the determination that the training error for the training batch is below the threshold, modifying the learning factor.

7. The system of claim 5 , wherein the fine-tuning of the HD-CNN comprises:
applying a temporal sparsity penalty term in evaluating each of the fine CNNs.

8. The system of claim 1 , wherein the dataset comprising the categorized data comprises categorized images.

9. A method comprising:
accessing, by one or more processors, a dataset comprising categorized data having a plurality of fine categories;
dividing, by the one or more processors, the dataset into a training set and a value set;
training, by the one or more processors, a first CNN model using the training set;
generating, by the one or more processors, a confusion matrix of the first CNN model using the value set;
applying, by the one or more processors, a clustering algorithm to the confusion matrix to determine, for each fine category, an associated coarse category of a plurality of coarse categories, each coarse category being associated with more than one fine category;
training, by the one or more processors, a base convolutional neural network (CNN) to distinguish between the coarse categories, the base CNN implemented by a processor of a machine;
training, by the one or more processors, a single fine CNN for each coarse category, the single fine CNN for said coarse category to distinguish between the fine categories associated with said coarse category;
receiving, by the one or more processors, a request to classify data;
using the base CNN, determining, by the one or more processors, a coarse category of the data;
using the single fine CNN for the determined coarse category, determining, by the one or more processors, a single fine category of the data; and
responsive to the request, transmitting, by the one or more processors, the single fine category of the data.

10. The method of claim 9 , wherein the clustering algorithm comprises an affinity propagation algorithm.

11. The method of claim 10 , further comprising obtaining low-dimensional feature representations for the fine categories using a Laplacian eigenmap.

12. The method of claim 10 , wherein the training of the fine CNN for each coarse category comprises:
training a second CNN model using the training set;
generating the fine CNN for each coarse category from the second CNN; and
training the fine CNN for each coarse category using a subset of the training set, the subset excluding data having fine categories not associated with said coarse category.

13. The method of claim 9 , further comprising:
combining the base CNN with each of the fine CNNs to form a hierarchically deep CNN (HD-CNN); and
fine-tuning the HD-CNN.

14. The method of claim 13 , wherein the fine-tuning of the HD-CNN includes:
beginning the fine-tuning with a learning factor;
training the HD-CNN by iterating over a series of training batches using the learning factor:
after each iteration, comparing a training error for the training batch to a threshold;
determining, based on the comparison, that the training error for the training batch is below the threshold; and
in response to the determination that the training error for the training batch is below the threshold, modifying the learning factor.

15. The method of claim 13 , wherein the fine-tuning of the HD-CNN includes:
applying a temporal sparsity penalty term in evaluating each of the fine CNNs.

16. The system of claim 1 , wherein:
the using of the base CNN to determine the coarse category of the data comprises:
providing an input to the base CNN; and
receiving a first output from the base CNN; and
the using of the fine CNN for the determined coarse category to determine the single fine category of the data comprises:
providing the input to the fine CNN; and
receiving a second output from the fine CNN.

17. The system of claim 1 , wherein:
the receiving of the request to classify data comprises receiving, from a computing device via a network, an image; and
the operations further comprise:
generating a listing template based on the single fine category; and
causing the listing template to be presented on the computing device.

18. A non-transitory machine-readable medium having instructions embodied thereon, the instructions executable by a processor of a machine to perform operations comprising:
accessing a dataset comprising categorized data having a plurality of fine categories;
dividing the dataset into a training set and a value set;
training a first CNN model using the training set;
generating a confusion matrix of the first CNN model using the value set;
applying a clustering algorithm to the confusion matrix to determine, for each fine category, an associated coarse category of a plurality of coarse categories, each coarse category being associated with more than one fine category;
training a base convolutional neural network (CNN) to distinguish between the coarse categories, the CNN i

Assignees

• Ebay Inc

Inventors

• Yan Zhicheng
• Piramuthu Robinson
• Jagadeesh Vignesh
• Di Wei
• Decoste Dennis

Classifications

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/0454

  Physics · mapped topic

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title