Subcategory-aware convolutional neural networks for object detection

Patent metadata

Field	Value
Publication number	US-9965719-B2
Application number	US-201615342823-A
Country	US
Kind code	B2
Filing date	Nov 3, 2016
Priority date	Nov 4, 2015
Publication date	May 8, 2018
Grant date	May 8, 2018

Abstract

A computer-implemented method for detecting objects by using subcategory-aware convolutional neural networks (CNNs) is presented. The method includes generating object region proposals from an image by a region proposal network (RPN) which utilizes subcategory information, and classifying and refining the object region proposals by an object detection network (ODN) that simultaneously performs object category classification, subcategory classification, and bounding box regression. The image is an image pyramid used as input to the RPN and the ODN. The RPN and the ODN each include a feature extrapolating layer to detect object categories with scale variations among the objects.

First claim

What is claimed is:

1. A computer-implemented method for detecting objects by using subcategory-aware convolutional neural networks (CNNs), the method comprising:
generating object region proposals from an image by a region proposal network (RPN) which utilizes subcategory information; and
classifying and refining the object region proposals by an object detection network (ODN) that simultaneously performs object category classification, subcategory classification, and bounding box regression,
wherein the RPN and the ODN each include a feature extrapolating layer to detect object categories with scale variations among the objects.

2. The method of claim 1 , wherein the image is an image pyramid used as input to the RPN and the ODN.

3. The method of claim 1 , wherein the RPN includes a subcategory convolution layer such that each filter in the convolution layer corresponds to an object subcategory.

4. The method of claim 3 , wherein the subcategory convolution layer outputs heats maps indicating a presence of subcategories at a specific location and a specific scale.

5. The method of claim 4 , wherein the heat maps are used to create a region of interest (RoI) generating layer for generating object candidates by thresholding the heat maps.

6. The method of claim 5 , wherein the RPN terminates at two layers, one layer that outputs softmax probability estimates over object subcategories and another layer that refines RoI location with a bounding box regressor.

7. A system for detecting objects by using subcategory-aware convolutional neural networks (CNNs), the system comprising:
a memory; and
a hardware computer processor operatively coupled to the memory, the processor being configured for:
generating object region proposals from an image by a region proposal network (RPN) which utilizes subcategory information; and
classifying and refining the object region proposals by an object detection network (ODN) that simultaneously performs object category classification, subcategory classification, and bounding box regression,
wherein the RPN and the ODN each include a feature extrapolating layer to detect object categories with scale variations among the objects.

8. The system of claim 7 , wherein the image is an image pyramid used as input to the RPN and the ODN.

9. The system of claim 7 , wherein the RPN includes a subcategory convolution layer such that each filter in the convolution layer corresponds to an object subcategory.

10. The system of claim 9 , wherein the subcategory convolution layer outputs heats maps indicating a presence of subcategories at a specific location and a specific scale.

11. The system of claim 10 , wherein the heat maps are used to create a region of interest (RoI) generating layer for generating object candidates by thresholding the heat maps.

12. The system of claim 11 , wherein the RPN terminates at two layers, one layer that outputs softmax probability estimates over object subcategories and another layer that refines RoI location with a bounding box regressor.

13. A non-transitory computer-readable storage medium comprising a computer-readable program for detecting objects by using subcategory-aware convolutional neural networks (CNNs), wherein the computer-readable program when executed on a computer, using a hardware computer processor coupled to the non-transitory computer-readable storage medium, causes the computer to perform the steps of:
generating object region proposals from an image by a region proposal network (RPN) which utilizes subcategory information; and
classifying and refining the object region proposals by an object detection network (ODN) that simultaneously performs object category classification, subcategory classification, and bounding box regression,
wherein the RPN and the ODN each include a feature extrapolating layer to detect object categories with scale variations among the objects.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the image is an image pyramid used as input to the RPN and the ODN.

15. The non-transitory computer-readable storage medium of claim 13 , wherein the RPN includes a subcategory convolution layer such that each filter in the convolution layer corresponds to an object subcategory.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the subcategory convolution layer outputs heats maps indicating a presence of subcategories at a specific location and a specific scale.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the heat maps are used to create a region of interest (RoI) generating layer for generating object candidates by thresholding the heat maps.

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Choi Wongun
• Lin Yuanqing
• Xiang Yu
• Savarese Silvio

Classifications

• G06T2207/30261

  Obstacle · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title