Method, medium, and system for image text localization and comparison

Patent metadata

Field	Value
Publication number	US-10970768-B2
Application number	US-201615349462-A
Country	US
Kind code	B2
Filing date	Nov 11, 2016
Priority date	Nov 11, 2016
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

Systems, methods, and computer program products for identifying a candidate product in an electronic marketplace based on a visual comparison between candidate product image visual text content and input query image visual text content. Unlike conventional optical character recognition (OCR) based systems, embodiments automatically localize and isolate portions of a candidate product image and an input query image that each contain visual text content, and calculate a visual similarity measure between the respective portions. A trained neural network may be re-trained to more effectively find visual text content by using the localized and isolated visual text content portions as additional ground truths. The visual similarity measure serves as a visual search result score for the candidate product. Any number of images of any number of candidate products may be compared to an input query image to enable text-in-image based product searching without resorting to conventional OCR techniques.

First claim

What is claimed is:

1. A method comprising:
analyzing, using one or more processors, a candidate product image using a machine learning model to identify candidate product image visual text content;
updating the machine learning model with the candidate product image visual text content;
analyzing an input query image using the updated machine learning model to identify input query image visual text content;
determining a visual similarity measure between the candidate product image visual text content and the input query image visual text content based on image signatures associated with each of the candidate product image and the input query image;
ranking a candidate product in a product list based on the visual similarity measure; and
causing presentation of the product list on a graphical user interface of a client device.

2. The method of claim 1 , wherein the updating the machine learning model further comprises:
generating augmented training content from the candidate product image by performing at least one of color littering, random scaling, mirror imaging, random cropping, or synthetic image generation; and
updating the machine learning model using the augmented training content.

3. The method of claim 1 , further comprising:
providing an image portion to a translator tool that generates an image of translated visual text content for another human language;
replacing an image portion visual text content with the image of the translated visual text content to produce a modified image; and
outputting the modified image.

4. The method of claim 1 , further comprising:
filling a form with visual text content from one of the input image query or the candidate product image, the form comprising at least one of: an item listing description, a shipping label; a receipt, or a text search query.

5. The method of claim 1 , further comprising:
comparing a reference image of one of a handwritten signature or a photo identification with a candidate image comprising a respective one of a second handwritten signature or a second photo identification provided for a transaction in the electronic marketplace.

6. The method of claim 5 , further comprising:
authorizing at least one of payment and product delivery based on the comparing.

7. The method of claim 1 , further comprising:
training the machine learning model offline with images of a plurality of products in an electronic marketplace; and
updating the machine learning model in substantially real time with an image of a new product provided to the electronic marketplace.

8. A non-transitory computer-readable storage medium having embedded therein a set of instructions which; when executed by one or more hardware-based processors of a computer, causes the computer to perform operations comprising:
analyzing a candidate product image using a machine learning model to identify candidate product image visual text content;
updating the machine learning model with the candidate product image visual text content;
analyzing an input query image using the updated machine learning model to identify input query image visual text content;
determining a visual similarity measure between the candidate product image visual text content and the input query image visual text content based on image signatures associated with each of the candidate product image and the input query image;
ranking the candidate product in a product list based on the visual similarity measure; and
causing presentation of the product list on a graphical user interface of a client device.

9. The medium of claim 8 , wherein updating the machine learning model further comprises:
generating augmented training content from the candidate product image by performing at least one of: color jittering, random scaling, mirror imaging, random cropping, or synthetic image generation; and
updating the machine learning model using the augmented training content.

10. The medium claim 8 , wherein the operations further comprise:
providing an image portion to a translator tool that generates an image of translated visual text content for another human language;
replacing an image portion visual text content with the image of the translated visual text content to produce a modified image; and
outputting the modified image.

11. The medium of claim 8 , wherein the operations further comprise:
filling a form with visual text content from one of the input image query and the candidate product image, the form comprising at least one of: an item listing description, a shipping label, a receipt, and a text search query.

12. The medium of claim 8 , wherein the operations further comprise:
comparing a reference image of one of a handwritten signature or a photo identification with a candidate image comprising a respective one of a second handwritten signature or a second photo identification provided for a transaction in an electronic marketplace.

13. The medium of claim 12 , wherein the operations further comprise:
authorizing at least one of payment and product delivery based on the comparing.

14. The medium of claim 8 , wherein the operations further comprise:
training the machine learning model offline with images of a plurality of products in an electronic marketplace; and
updating the machine learning model in substantially real time with an image of a new product provided to the electronic marketplace.

15. A system comprising:
a memory comprising instructions; and
one or more hardware-based computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
analyzing a candidate product image using a machine learning model to identify candidate product image visual text content;
updating the machine learning model with the candidate product image visual text content;
analyzing an input query image using the updated machine learning model to identify input query image visual text content;
determining a visual similarity measure between the candidate product image visual text content and the input query image visual text content based on image signatures associated with each of the candidate product image and the input query image;
ranking the candidate product in a product list based on the visual similarity measure; and
causing presentation of the product list on a graphical user interface of a client device.

16. The system of claim 15 , wherein updating the machine learning model further comprises:
generating augmented training content from the candidate product image by performing at least one of: color jittering, random scaling, mirror imaging, random cropping, or synthetic image generation; and
updating the machine learning model using the augmented training content.

17. The system of claim 15 , wherein the operations further comprise:
providing an image portion to a translator tool that generates an image of translated visual text content for another human language;
replacing an image portion visual text content with the image of the translated visual text content to produce a modified image; and
outputting the modified image.

18. The system of claim 15 , wherein the operations further comprise:
filling a form with visual text content from one of the input image query and the candidate product image, the form comprising at least one of: an item listing description, a shipping label, a receipt, or a text search query.

19. The system of claim 15 , wherein the o

Assignees

• Ebay Inc

Inventors

• Zheng Shuai
• Piramuthu Robinson

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title