Image processing and matching

Patent metadata

Field	Value
Publication number	US-10528618-B2
Application number	US-201715641108-A
Country	US
Kind code	B2
Filing date	Jul 3, 2017
Priority date	Aug 12, 2014
Publication date	Jan 7, 2020
Grant date	Jan 7, 2020

Abstract

A configured machine performs image matching and retrieval of natural images that may depict logos. The machine generates and uses color-localized spatial masks, which may be computationally less expensive than spatial verification techniques. Key points are detected within images that form a reference database of images. Local masks are defined by the machine around each key point based on the scale and orientation of the key point. To utilize color information presented in logo images, ordered color histograms may be extracted by the machine from locally masked regions of each image. A cascaded index may then be constructed for both visual descriptors and color histograms. For faster matching, the cascaded index maps the visual descriptors and color histograms to a list of relevant or similar images. This list may then be ranked to generate relevant matches for an input query image.

First claim

What is claimed is:

1. A system allowing for logo image matching, comprising:
a processor;
a network interface device; and
a storage device having stored thereon computer instructions executable by the system to cause the system to perform operations comprising:
receiving a first real-world image comprising a visual depiction of one or more physical objects;
identifying, within the first real-world image, a graphical logo portrayed on a first physical object of the one or more physical objects, the identifying comprising:
selecting a portion of the first image;
capturing spatial color information for the portion of the first image;
calculating a color-localized image representation for the portion of the first image based on the spatial color information; and
performing a two-level indexing process of the first image using both the color-localized image representation for the portion of the first image and the spatial color information for the portion of the first image; and
based on identifying the graphical logo, responding to a user request.

2. The system of claim 1 , wherein the identifying further comprises:
generating a first index that correlates first clusters of local feature descriptors with reference images of a plurality of reference images, the local feature descriptors generated from the plurality of reference images;
generating a second index that correlates second clusters of local color descriptors with reference images of the plurality of reference images, one of the local color descriptors generated from a reference image among the set of reference images based on color histograms generated from multiple regions that correspond to a key point within the reference image;
generating a third index that correlates descriptor pairs with reference images from the plurality of reference images, the descriptor pairs representing respective co-occurrences of a first cluster of the local feature descriptors with a second cluster of the local color descriptors;
wherein the identifying is based on the third index correlating a descriptor pair among the descriptor pairs with the reference image.

3. The system of claim 1 , wherein the user request is a query submitted via an Internet enabled computing device.

4. The system of claim 3 , wherein the query specifies a sample logo image as part of the query and wherein the query seeks other images having similarity to the sample logo image.

5. The system of claim 1 , wherein responding to the user request comprises causing a display of the first-real world image.

6. The system of claim 1 , wherein the operations further comprise:
searching for the graphical logo in a plurality of real-world images including the first real-world image.

7. A method, comprising:
receiving, at a computer system, a first real-world image comprising a visual depiction of one or more physical objects;
the computer system identifying, within the first real-world image, a graphical logo portrayed on a first physical object of the one or more physical objects, the identifying comprising:
selecting a portion of the first image;
capturing spatial color information for the portion of the first image;
calculating a color-localized image representation for the portion of the first image based on the spatial color information; and
performing a two-level indexing process of the first image using both the color-localized image representation for the portion of the first image and the spatial color information for the portion of the first image; and
based on identifying the graphical logo, responding to a user request.

8. The method of claim 7 , wherein the first physical object is a manufactured article having the graphical logo affixed thereto.

9. The method of claim 8 , wherein the manufactured article is a piece of clothing.

10. The method of claim 7 , wherein the manufactured article is a household good.

11. The method of claim 7 , wherein the identifying further comprises:
generating a first index that correlates first clusters of local feature descriptors with reference images of a plurality of reference images, the local feature descriptors generated from the plurality of reference images;
generating a second index that correlates second clusters of local color descriptors with reference images of the plurality of reference images, one of the local color descriptors generated from a reference image among the set of reference images based on color histograms generated from multiple regions that correspond to a key point within the reference image;
generating a third index that correlates descriptor pairs with reference images from the plurality of reference images, the descriptor pairs representing respective co-occurrences of a first cluster of the local feature descriptors with a second cluster of the local color descriptors;
wherein the identifying is based on the third index correlating a descriptor pair among the descriptor pairs with the reference image.

12. The method of claim 7 , wherein the user request is made by a user from a mobile computing device.

13. The method of claim 7 , wherein a bag-of-words (BoW) approach is used to generate the color-localized image representation.

14. The method of claim 7 , further comprising detecting one or more key points of the first image and using those key points to compare the first image to a plurality of real-world images stored in a database, the plurality of real-world images corresponding to a plurality of graphical logos.

15. The method of claim 7 , wherein the graphical logo is a trademarked logo of a corporation.

16. A non-transitory computer-readable medium having instructions stored thereon that are executable by a computer system to cause the computer system to perform operations comprising:
receiving a first real-world image comprising a visual depiction of one or more physical objects;
identifying, within the first real-world image, a graphical logo portrayed on a first physical object of the one or more physical objects, the identifying comprising:
selecting a portion of the first image;
capturing spatial color information for the portion of the first image;
calculating a color-localized image representation for the portion of the first image based on the spatial color information
performing a two-level indexing process of the first image using both the color-localized image representation for the portion of the first image and the spatial color information for the portion of the first image; and
based on identifying the graphical logo, responding to a user request.

17. The non-transitory computer-readable medium of claim 16 , wherein the user request specifies a sample logo image as part of the request.

18. The non-transitory computer-readable medium of claim 16 , wherein a bag-of-words (BoW) approach is used to generate the color-localized image representation.

19. The non-transitory computer-readable medium of claim 16 , wherein the color-localized image representation includes feature descriptors represented as multidimensional vectors.

20. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise identifying key points in the first image based on color histogram information, and wherein the identifying is based on analysis of the key points.

Assignees

• Paypal Inc

Inventors

• Di Wei
• Jagadeesh Vignesh
• Piramuthu Robinson
• Pandey Rohit
• Bhardwaj Anurag

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06F16/583 Primary

  using metadata automatically derived from the content · CPC title

• G06F16/5838

  using colour · CPC title

• G06K9/6211

  Physics · mapped topic