Generic object detection in images

Patent metadata

Field	Value
Publication number	US-2016104058-A1
Application number	US-201514617909-A
Country	US
Kind code	A1
Filing date	Feb 9, 2015
Priority date	Oct 9, 2014
Publication date	Apr 14, 2016
Grant date	—

Abstract

Neural networks for object detection in images are used with a spatial pyramid pooling (SPP) layer. Using the SPP network structure, a fixed-length representation is generated regardless of image size and scale. The feature maps are computed from the entire image once, and the features are pooled in arbitrary regions (sub-images) to generate fixed-length representations for training the detectors. Thus, repeated computation of the convolutional features is avoided while accuracy is enhanced.

First claim

What is claimed is:

1 . A method to perform object detection in an image, the method comprising:
receiving an input image; generating feature maps by one or more filters on a convolutional layer of a neural network processing the input image; spatially pooling responses of each filter at a spatial pyramid pooling (SPP) layer; providing outputs of the SPP layer to a fully-connected layer as fixed dimensional vectors; and training a classifier to detect one or more objects in the input image based on the fixed dimensional vectors received at the fully-connected layer.

2 . The method of claim 1 , wherein generating the feature maps comprises:
employing sliding filters at one or more convolutional layers configured to accept inputs of arbitrary sizes and provide outputs that approximate an aspect ratio of the inputs.

3 . The method of claim 1 , wherein spatially pooling the responses of each filter comprises:
pooling the responses of each filter in a plurality of spatial bins; and generating a multi-dimensional output vector, wherein a number of dimensions of the output vector is based on a number of the plurality of spatial bins multiplied by a number of filters in a last convolutional layer.

4 . The method of claim 1 , further comprising:
applying the SPP on each candidate window of the feature maps to pool a fixed-length representation of each candidate window.

5 . The method of claim 1 , further comprising:
resizing the image following feature extraction.

6 . The method of claim 1 , wherein the outputs of the SPP layer are representations of each window such that the classifier is trained for each category of the representations.

7 . The method of claim 1 , wherein training the classifier comprises:
employing ground-truth windows to generate positive samples; and identifying negative samples based on an overlap with a positive window below a first predefined threshold.

8 . The method of claim 7 , further comprising:
removing a negative sample that overlaps with another negative sample above a second predefined threshold.

9 . The method of claim 1 , further comprising:
in a test mode, scoring candidate windows through the classifier.

10 . The method of claim 9 , further comprising:
employing non-maximum suppression with a predefined threshold on the scored candidate windows.

11 . A computing device to perform object detection in an image, the computing device comprising:
an input module configured to receive an input image through one or more of a wired or wireless communication; a memory configured to store instructions; and a processor coupled to the memory and the input module, the processor executing an image processing application, wherein the image processing application is configured to: generate feature maps by employing one or more sliding filters on a convolutional layer of a neural network processing the input image; spatially pool responses of each filter in a plurality of spatial bins at a spatial pyramid pooling (SPP) layer; provide outputs of the SPP layer to a fully-connected layer as fixed dimensional vectors; and train a classifier to detect one or more objects in the input image based on the fixed dimensional vectors received at the fully-connected layer.

12 . The computing device of claim 11 , wherein the one or more sliding filters are activated by semantic content.

13 . The computing device of claim 11 , wherein the feature maps are generated once from the entire input image at one or more scales.

14 . The computing device of claim 11 , wherein the image processing application is further configured to:
resize the image; generate the feature maps for each scale; and combine features for each scale by pooling the features channel-by-channel.

15 . The computing device of claim 11 , wherein the SPP layer comprises a 4-level spatial pyramid of 1×1, 2×2, 3×3, and 6×6 configuration that yields a total of 50 spatial bins.

16 . The computing device of claim 11 , wherein the image processing application is further configured to:
fine-tune the fully-connected layer by initializing weights of the fully-connected layer, performing a first training using a first learning rate and performing a second training using a refined second learning rate.

17 . The computing device of claim 11 , wherein the image processing application is further configured to:
post-process prediction windows using bounding-box regression, wherein features used for regression are pooled features from the convolution layer.

18 . A computer-readable memory device with instructions stored thereon to perform object detection in an image, the instructions comprising:
receiving an input image; generating feature maps by one or more filters on a convolutional layer of a first neural network processing the input image; extracting window-wise features from regions of deep convolutional feature maps; performing a selective search to generate a predefined number of candidate windows per image; spatially pooling responses of candidate windows at a spatial pyramid pooling (SPP) layer; providing outputs of the SPP layer to a fully-connected layer as fixed dimensional vectors; and training a classifier to detect one or more objects in the input image based on the fixed dimensional vectors received at the fully-connected layer.

19 . The computer-readable memory device of claim 18 , wherein the instructions further comprise:
resizing the input image such that min (w; h)=s, where w is a width of the image, h is a height of the image, and s represents a predefined scale for the image.

20 . The computer-readable memory device of claim 19 , wherein the instructions further comprise:
pre-training the first neural network and a second neural network with different random initializations; scoring candidate windows on a test image through the first neural network and the second neural network; and performing non-maximum suppression on a union of two sets of candidate windows with their respective scores; and selecting a window with higher score from the first neural network or the second neural network for the detection of the object.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• He Kaiming
• Sun Jian
• Zhang Xiangyu

Classifications

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title

• G06K9/6255

  Physics · mapped topic

• G06K9/66 Primary

  Physics · mapped topic

• G06F17/30247

  Physics · mapped topic