Hybrid detection recognition system

Patent metadata

Field	Value
Publication number	US-10922353-B2
Application number	US-201916265149-A
Country	US
Kind code	B2
Filing date	Feb 1, 2019
Priority date	Aug 14, 2013
Publication date	Feb 16, 2021
Grant date	Feb 16, 2021

Abstract

A system and method for determining an object or product represented in an image is disclosed. The system receives a first image, determines a region of interest in the first image, determines a classification score for the region of interest using a convolutional neural network that assigns the region of interest the classification score corresponding to a class, and identifies a first product in the first image based on the classification score.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more processors, a first image;
determining, by the one or more processors, a first region of interest and a second region of interest in the first image;
determining, by the one or more processors, a first classification score for the first region of interest and a second classification score for the second region of interest using a convolutional neural network, the convolutional neural network assigning the first region of interest the first classification score corresponding to a class and the second region of interest the second classification score corresponding to the class;
determining, by the one or more processors, whether the first region of interest and the second region of interest share a similar spatial location;
responsive to determining that the first region of interest and the second region of interest share the similar spatial location, combining, by the one or more processors, the first classification score and the second classification score to determine a result class for the similar spatial location; and
identifying, by the one or more processors, a first product in the first image based on the combined classification score.

2. The method of claim 1 , wherein the result class includes a product class associated with a second product, and wherein the combined classification score corresponding to the product class indicates a likelihood that the first product in the first image is the second product associated with the product class.

3. The method of claim 1 , wherein:
the convolutional neural network is trained on a training data of images for inferring one or more rules;
the convolutional neural network uses the one or more inferred rules for classifying and assigning the first region of interest the first classification score corresponding to the class and the second region of interest the second classification score corresponding to the class; and
the training data of images belong to the class.

4. The method of claim 1 , wherein determining the first region of interest comprises:
extracting a first feature of the first image;
determining whether the first feature of the first image matches a second feature of a second image;
determining whether a shape formed by the first feature of the first image is geometrically consistent with a shape formed by the second feature of the second image; and
responsive to determining that the shape formed by the first feature of the first image is geometrically consistent with the shape formed by the second feature of the second image, identifying the shape formed by the first feature of the first image as the first region of interest.

5. The method of claim 1 , wherein determining the first classification score for the first region of interest includes determining a plurality of first classification scores corresponding to a plurality of classes, and the method further comprises:
matching the first region of interest in the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image;
adjusting the plurality of first classification scores based on the matching score; and
identifying the first product in the first image based on the plurality of adjusted first classification scores.

6. The method of claim 1 , further comprising:
identifying the result class for the first image based on the combined classification score;
determining whether the result class is a product class; and
responsive to determining that the result class is not the product class,
matching the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image; and
identifying the first product in the first image based on the matching score.

7. The method of claim 6 , wherein determining the first region of interest and the second region of interest comprises:
determining a plurality of regions of interest that share the similar spatial location in the first image;
ranking the plurality of regions of interest; and
selecting, from the plurality of regions of interest, a predetermined number of regions of interest for the similar spatial location based on the ranking.

8. A system comprising:
one or more processors; and
a memory, the memory storing instructions, which when executed cause the one or more processors to:
receive a first image;
determine a first region of interest and a second region of interest in the first image;
determine a first classification score for the first region of interest and a second classification score for the second region of interest using a convolutional neural network, the convolutional neural network assigning the first region of interest the first classification score corresponding to a class and the second region of interest the second classification score corresponding to the class;
determine whether the first region of interest and the second region of interest share a similar spatial location;
responsive to a determination that the first region of interest and the second region of interest share the similar spatial location, combine the first classification score and the second classification score to determine a result class for the similar spatial location; and
identify a first product in the first image based on the combined classification score.

9. The system of claim 8 , wherein the result class includes a product class associated with a second product, and wherein the combined classification score corresponding to the product class indicates a likelihood that the first product in the first image is the second product associated with the product class.

10. The system of claim 8 , wherein:
the convolutional neural network is trained on a training data of images for inferring one or more rules;
the convolutional neural network uses the one or more inferred rules for classifying and assigning the first region of interest the first classification score corresponding to the class and the second region of interest the second classification score corresponding to the class; and
the training data of images belong to the class.

11. The system of claim 8 , wherein to determine the first region of interest, the instructions further cause the one or more processors to:
extract a first feature of the first image;
determine whether the first feature of the first image matches a second feature of a second image;
determine whether a shape formed by the first feature of the first image is geometrically consistent with a shape formed by the second feature of the second image; and
responsive to determining that the shape formed by the first feature of the first image is geometrically consistent with the shape formed by the second feature of the second image, identify the shape formed by the first feature of the first image as the first region of interest.

12. The system of claim 8 , wherein to determine the first classification score for the first region of interest, the instructions cause the one or more processors to determine a plurality of first classification scores corresponding to a plurality of classes, and the instructions further cause the one or more processors to:
match the first region of interest in the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image;
adjust the plurality of first classification scores based on the matching score; and
identify the first product in the first image based on the plurality of adjusted first classification scores.

Assignees

• Kwon Junghyun
• Narasimha Ramya
• Schwartz Edward L
• Mcfarland Max
• Savarese Silvio
• Berkner Kathrin
• Ricoh Co Ltd

Inventors

• Kwon Junghyun
• Narasimha Ramya
• Schwartz Edward L
• Mcfarland Max
• Savarese Silvio
• Berkner Kathrin

Classifications

• G06Q30/0201 Primary

  Market modelling; Market analysis; Collecting market data · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06F16/51 Primary

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F18/254

  of classification results, e.g. of results related to same input data · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title