Hybrid detection recognition system

Patent metadata

Field	Value
Publication number	US-10242036-B2
Application number	US-201615199553-A
Country	US
Kind code	B2
Filing date	Jun 30, 2016
Priority date	Aug 14, 2013
Publication date	Mar 26, 2019
Grant date	Mar 26, 2019

Abstract

A system and method for determining an object or product represented in an image is disclosed. The system receives a first image, determines a region of interest in the first image, determines a classification score for the region of interest using a convolutional neural network that assigns the region of interest the classification score corresponding to a class, and identifies a first product in the first image based on the classification score.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more processors, a first image;
determining, by the one or more processors, a plurality of regions of interest that share a similar spatial location in the first image;
ranking, by the one or more processors, the plurality of regions of interest;
selecting, from the plurality of regions of interest, a predetermined number of regions of interest for the similar spatial location based on the ranking;
determining, by the one or more processors, classification scores for the predetermined number of regions of interest using a convolutional neural network, the convolutional neural network assigning the predetermined number of regions of interest the classification scores corresponding to a product class;
matching, by the one or more processors, the predetermined number of regions of interest in the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image;
adjusting, by the one or more processors, the classification scores based on the matching score; and
identifying, by the one or more processors, a first product in the first image based on the adjusted classification scores.

2. The method of claim 1 , wherein the product class is associated with a second product, and wherein the classification scores corresponding to the product class indicate a likelihood that the first product in the first image is the second product associated with the product class.

3. The method of claim 1 , further comprising:
determining a first classification score for a first region of interest in the first image;
determining a second classification score for a second region of interest in the first image;
determining whether the first region of interest and the second region of interest share the similar spatial location; and
responsive to the first region of interest and the second region of interest sharing the similar spatial location, combining the first classification score and the second classification score to determine a result class for the similar spatial location.

4. The method of claim 1 , wherein matching the predetermined number of regions of interest in the first image to the indexed images using the model-based features comprises:
extracting a first feature of the first image;
determining whether the first feature of the first image matches a second feature of the second image;
determining whether a shape formed by the first feature of the first image is geometrically consistent with a shape formed by the second feature of the second image; and
responsive to determining that the shape formed by the first feature of the first image is geometrically consistent with the shape formed by the second feature of the second image, identifying the shape formed by the first feature of the first image as one of the predetermined number of regions of interest.

5. A system comprising:
one or more processors; and
a memory, the memory storing instructions, which when executed cause the one or more processors to:
receive a first image;
determine a plurality of regions of interest that share a similar spatial location in the first image;
rank the plurality of regions of interest;
select, from the plurality of regions of interest, a predetermined number of regions of interest for the similar spatial location based on the ranking;
determine classification scores for the predetermined number of regions of interest using a convolutional neural network, the convolutional neural network assigning the predetermined number of regions of interest the classification scores corresponding to a product class;
match the predetermined number of regions of interest in the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image;
adjust the classification scores based on the matching score; and
identify a first product in the first image based on the adjusted classification scores.

6. The system of claim 5 , wherein the product class is associated with a second product, and wherein the classification scores corresponding to the product class indicate a likelihood that the first product in the first image is the second product associated with the product class.

7. The system of claim 5 , wherein the instructions further cause the one or more processors to:
determine a first classification score for a first region of interest in the first image;
determine a second classification score for a second region of interest in the first image;
determine whether the first region of interest and the second region of interest share the similar spatial location; and
responsive to the first region of interest and the second region of interest sharing the similar spatial location, combine the first classification score and the second classification score to determine a result class for the similar spatial location.

8. The system of claim 5 , wherein to match the predetermined number of regions of interest in the first image to the indexed images using the model-based features, the instructions further cause the one or more processors to:
extract a first feature of the first image;
determine whether the first feature of the first image matches a second feature of the second image;
determine whether a shape formed by the first feature of the first image is geometrically consistent with a shape formed by the second feature of the second image; and
responsive to determining that the shape formed by the first feature of the first image is geometrically consistent with the shape formed by the second feature of the second image, identify the shape formed by the first feature of the first image as one of the predetermined number of regions of interest.

9. A computer program product comprising a non-transitory computer readable medium storing a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:
receive a first image;
determine a plurality of regions of interest that share a similar spatial location in the first image;
rank the plurality of regions of interest;
select, from the plurality of regions of interest, a predetermined number of regions of interest for the similar spatial location based on the ranking;
determine classification scores for the predetermined number of regions of interest using a convolutional neural network, the convolutional neural network assigning the predetermined number of regions of interest the classification scores corresponding to a product class;
match the predetermined number of regions of interest in the first image to indexed images using model-based features to determine a second image and a matching score associated with the second image;
adjust the classification scores based on the matching score; and
identify a first product in the first image based on the adjusted classification scores.

10. The computer program product of claim 9 , wherein the product class is associated with a second product, and wherein the classification scores corresponding to the product class indicate a likelihood that the first product in the first image is the second product associated with the product class.

11. The computer program product of claim 9 , wherein the computer readable program when executed on the computer further causes the computer to:
determine a first classification score for a first region of interest in the first image;
determine a second classification score for a second region of interest in the first image;
determine whether the first region of interest and the second region of i

Assignees

• Kwon Junghyun
• Narasimha Ramya
• Schwartz Edward L
• Mcfarland Max
• Savarese Silvio
• Berkner Kathrin
• Ricoh Co Ltd

Inventors

• Kwon Junghyun
• Narasimha Ramya
• Schwartz Edward L
• Mcfarland Max
• Savarese Silvio
• Berkner Kathrin

Classifications

• G06Q30/0201 Primary

  Market modelling; Market analysis; Collecting market data · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06F16/51

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F18/254

  of classification results, e.g. of results related to same input data · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title