Machine learned single image icon identification

Patent metadata

Field	Value
Publication number	US-10970577-B1
Application number	US-201816136998-A
Country	US
Kind code	B1
Filing date	Sep 20, 2018
Priority date	Sep 29, 2017
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

Systems, devices, media, and methods are presented for graphical icon identification within an image or video stream. The systems and methods receive an image including a graphical icon. The systems and methods identify a set of proposed regions of the image, at least one proposed region of the set of proposed regions containing the graphical icon and extract a set of semantic features for each proposed region of the set of proposed regions. Based on the set of semantic features of the set of proposed regions, the systems and methods identify a set of proposed icons corresponding to the graphical icon included in the image and determine a match between the graphical icon and at least one proposed icon of the set of proposed icons.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more processors, an image including a graphical icon;
identifying, by the one or more processors, a set of proposed regions of the image, at least one proposed region of the set of proposed regions containing the graphical icon;
extracting a set of semantic features for each proposed region of the set of proposed regions;
based on the set of semantic features of the set of proposed regions, identifying, by the one or more processors, a set of proposed icons corresponding to the graphical icon included in the image; and
determining a match between the graphical icon and at least one proposed icon of the set of proposed icons, the determining of the match comprising performing a pairwise comparison of visually distinct aspects of the graphical icon, included in the image, and each proposed icon of the set of proposed icons that are identified based on the set of semantic features.

2. The method of claim 1 , wherein identifying the set of proposed icons further comprises:
projecting the set of semantic features into an n-dimensional metric space mapping a plurality of semantic features to a plurality of icons;
determining a subset of icons, of the plurality of icons, within the n-dimensional metric space at a position corresponding to the set of semantic features proposed region including the graphical icon; and
selecting at least a portion of the subset of icons for inclusion in the set of proposed icons.

3. The method of claim 2 , wherein selecting the portion of the subset of icons for inclusion in the set of proposed icons further comprises:
determining a first numerical representation of semantic features corresponding to the graphical icon within the n-dimensional metric space;
determining a set of second numerical representations of semantic features of the subset of icons within the n-dimensional metric space; and
identifying the portion of the subset of icons for inclusion in the set of proposed icons based on a comparison of the first numerical representation and the set of second numerical representations.

4. The method of claim 3 , wherein comparing the first numerical representation and the set of second numerical representations further comprises:
identifying a distance threshold for the set of proposed icons, the distance threshold representing a maximum difference between the first numerical representation and the set of second numerical representations; and
selecting icons of the subset of icons associated with a distance below the distance threshold, the selected icons being included in the set of proposed icons.

5. The method of claim 1 , wherein determining the match between the graphical icon and the at least one proposed icon further comprises:
verifying the match between the graphical icon and the at least one proposed icon by generating a confidence score based on matches for one or more of geometric transformations, complementary feature embedding, and textual similarities among the graphical icon and the at least one proposed icon.

6. The method of claim 1 , wherein determining the match between the graphical icon and the at least one proposed icon further comprises:
determining similarities of semantic features of the graphical icon and the set of proposed icons; and
based on the similarities of semantic features, identifying a positive pair between the graphical icon and the at least one proposed icon.

7. The method of claim 1 , wherein the set of proposed icons is a first set of proposed icons, the method further comprising:
determining negative pairs for the graphical icon and each proposed icon of the first set of proposed icons;
generating a request for a second set of proposed icons by expanding the first set of proposed icons from a first number of icons to a second number of icons; and
determining the match between the graphical icon and the at least one proposed icon by determining a positive pair from comparison of the graphical icon and the at least one proposed icon within the second set of proposed icons.

8. A system comprising:
one or more processors; and
a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving an image including a graphical icon;
identifying a set of proposed regions of the image, at least one proposed region of the set of proposed regions containing the graphical icon;
extracting a set of semantic features for each proposed region of the set of proposed regions;
based on the set of semantic features of the set of proposed regions, identifying a set of proposed icons corresponding to the graphical icon included in the image; and
determining a match between the graphical icon and at least one proposed icon of the set of proposed icons, the determining of the match comprising performing a pairwise comparison of visually distinct aspects of the graphical icon, included in the image, and each proposed icon of the set of proposed icons that are identified based on the set of semantic features.

9. The system of claim 8 , wherein identifying the set of proposed icons further comprises:
projecting the set of semantic features into an n-dimensional metric space mapping a plurality of semantic features to a plurality of icons;
determining a subset of icons, of the plurality of icons, within the n-dimensional metric space at a position corresponding to the set of semantic features proposed region including the graphical icon; and
selecting at least a portion of the subset of icons for inclusion in the set of proposed icons.

10. The system of claim 9 , wherein selecting the portion of the subset of icons for inclusion in the set of proposed icons further comprises:
determining a first numerical representation of semantic features corresponding to the graphical icon within the n-dimensional metric space;
determining a set of second numerical representations of semantic features of the subset of icons within the n-dimensional metric space; and
identifying the portion of the subset of icons for inclusion in the set of proposed icons based on a comparison of the first numerical representation and the set of second numerical representations.

11. The system of claim 10 , wherein comparing the first numerical representation and the set of second numerical representations further comprises:
identifying a distance threshold for the set of proposed icons, the distance threshold representing a maximum difference between the first numerical representation and the set of second numerical representations; and
selecting icons of the subset of icons associated with a distance below the distance threshold, the selected icons being included in the set of proposed icons.

12. The system of claim 8 , wherein determining the match between the graphical icon and the at least one proposed icon further comprises:
verifying the match between the graphical icon and the at least one proposed icon by generating a confidence score based on matches for one or more of geometric transformations, complementary feature embedding, and textual similarities among the graphical icon and the at least one proposed icon.

13. The system of claim 8 , wherein determining the match between the graphical icon and the at least one proposed icon further comprises:
determining similarities of semantic features of the graphical icon and the set of proposed icons; and
based on the similarities of semantic features, identifying a positive pair between the graphical icon and the at least on

Assignees

• Snap Inc

Inventors

• Song Meng
• Agrawal Harsh
• Wang Xiaoyu
• Huang Xuan
• Brendel William

Classifications

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title