Identifying objects in images

Patent metadata

Field	Value
Publication number	US-9613297-B1
Application number	US-201514980494-A
Country	US
Kind code	B1
Filing date	Dec 28, 2015
Priority date	Dec 4, 2013
Publication date	Apr 4, 2017
Grant date	Apr 4, 2017

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for identifying objects in images. One of the methods includes receiving an input image; down-sampling the input image to generate a second image; generating a respective first score for each of the plurality of object categories; selecting an initial patch of the input image; generating a respective second score for each of the plurality of object categories; and generating a respective third score for each of the plurality of object categories from the first scores and the second scores, wherein the respective third score for each of the plurality of object categories represents a likelihood that the input image contains an image of an object belonging to the object category.

First claim

What is claimed is:

1. A method comprising:
receiving, by an image classification system, an input image having a first resolution;
down-sampling, by the image classification system, the input image to generate a second image having a second, lower resolution;
selecting, by the image classification system, an initial, first-resolution patch of the input image;
processing, by the image classification system, the second image using a low-resolution neural network, wherein the low-resolution neural network is configured to process the second image to generate a plurality of first scores, wherein each first score corresponds to a respective object category from a predetermined set of object categories, and wherein each first score represents a respective likelihood that the second image contains an image of an object belonging to the corresponding object category;
processing, by the image classification system, the initial patch of the input image using an initial patch neural network, wherein the initial patch neural network is configured to process the initial patch of the input image to generate a plurality of second scores, wherein each second score corresponds to a respective object category from the predetermined set of object categories, and wherein each second score represents a respective likelihood that the initial patch contains an image of an object belonging to the corresponding object category; and
generating, by the image classification system, a respective third score corresponding to each object category from the predetermined set of object categories from the first scores and the second scores, wherein each third score represents a respective likelihood that the input image contains an image of an object belonging to the corresponding object category.

2. The method of claim 1 , wherein generating the third scores comprises:
for each object category, combining the first score corresponding to the object category and the second score corresponding to the object category to generate the third score corresponding to the object category.

3. The method of claim 2 , wherein combining the first score corresponding to the object category and the second score corresponding to the object category comprises computing a geometric mean of the first score and the second score.

4. The method of claim 1 , wherein the low-resolution neural network is configured to:
process the second image to generate a plurality of features of the second image; and
process the plurality of features of the second image to generate the plurality of first scores.

5. The method of claim 4 , wherein selecting, by the image classification system, the initial, first-resolution patch of the input image comprises:
generating coordinates of a reference point in the input image by processing the plurality of features of the second image and the first scores using a patch locator neural network; and
selecting a patch of the input image that is centered at the reference point as the initial, first-resolution patch of the input image.

6. The method of claim 4 ,
wherein the low-resolution neural network comprises a plurality of layers,
wherein each of the layers is configured to, during the processing of the second image using the low-resolution neural network, receive a respective input at each of the plurality of layers and producing a respective output based on a set of parameters of the layer, and
wherein the features of the second image are the output of a pre-determined layer of the plurality of layers.

7. The method of claim 4 , further comprising:
processing the features of the second image, the first scores, and the second scores using an additional patch locator neural network to select an additional patch of the input image at the first resolution; and
processing the additional patch using an additional patch neural network, wherein the additional patch neural network is configured to process the additional patch to generate a plurality of fourth scores, each fourth score corresponding to a respective object category of the predetermined set of object categories.

8. The method of claim 7 , wherein generating the third scores comprises:
for each object category, combining the first score corresponding to the object category, the second score corresponding to the object category, and the fourth score corresponding to the object category to generate the third score for the object category.

9. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving, by an image classification system, an input image having a first resolution;
down-sampling, by the image classification system, the input image to generate a second image having a second, lower resolution;
selecting, by the image classification system, an initial, first-resolution patch of the input image;
processing, by the image classification system, the second image using a low-resolution neural network, wherein the low-resolution neural network is configured to process the second image to generate a plurality of first scores, wherein each first score corresponds to a respective object category from a predetermined set of object categories, and wherein each first score represents a respective likelihood that the second image contains an image of an object belonging to the corresponding object category;
processing, by the image classification system, the initial patch of the input image using an initial patch neural network, wherein the initial patch neural network is configured to process the initial patch of the input image to generate a plurality of second scores, wherein each second score corresponds to a respective object category from the predetermined set of object categories, and wherein each second score represents a respective likelihood that the initial patch contains an image of an object belonging to the corresponding object category; and
generating, by the image classification system, a respective third score corresponding to each object category from the predetermined set of object categories from the first scores and the second scores, wherein each third score represents a respective likelihood that the input image contains an image of an object belonging to the corresponding object category.

10. The system of claim 9 , wherein generating the third scores comprises:
for each object category, combining the first score corresponding to the object category and the second score corresponding to the object category to generate the third score corresponding to the object category.

11. The system of claim 10 , wherein combining the first score corresponding to the object category and the second score corresponding to the object category comprises computing a geometric mean of the first score and the second score.

12. The system of claim 9 , wherein the low-resolution neural network is configured to:
process the second image to generate a plurality of features of the second image; and
process the plurality of features of the second image to generate the plurality of first scores.

13. The system of claim 12 , wherein selecting, by the image classification system, the initial, first-resolution patch of the input image comprises:
generating coordinates of a reference point in the input image by processing the plurality of features of the second image and the first scores using a patch locator neural network; and
selecting a patch of the input image that is centered at the reference point as the initial, first-resolution patch of the input

Assignees

• Google Inc

Inventors

• Ranzato Marc'Aurelio

Classifications

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title

• G06V10/255 Primary

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06F18/285

  Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system · CPC title

• G06F18/241

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title

• G06N3/045

  Combinations of networks · CPC title