Method, medium, and system for intelligent online personal assistant with image text localization

Patent metadata

Field	Value
Publication number	US-2025117839-A1
Application number	US-202418986611-A
Country	US
Kind code	A1
Filing date	Dec 18, 2024
Priority date	Nov 11, 2016
Publication date	Apr 10, 2025
Grant date	—

Abstract

Systems, methods, and computer program products for identifying a candidate product in an electronic marketplace based on a visual comparison between candidate product image visual text content and input query image visual text content. Unlike conventional optical character recognition (OCR) based systems, embodiments automatically localize and isolate portions of a candidate product image and an input query image that each contain visual text content, and calculate a visual similarity measure between the respective portions. A trained neural network may be re-trained to more effectively find visual text content by using the localized and isolated visual text content portions as additional ground truths. The visual similarity measure serves as a visual search result score for the candidate product. Any number of images of any number of candidate products may be compared to an input query image to enable text-in-image based product searching without resorting to conventional OCR techniques.

First claim

1 - 20 . (canceled)

21 . A method comprising:
receiving an input query image; analyzing the input query image using a machine learning model to identify input query image visual text content; determining a visual similarity measure between a candidate product image visual text content and the input query image visual text content based on image signatures associated with the candidate product image and the input query image; recommending a candidate product based on the visual similarity measure; and causing presentation of the recommended candidate product on a graphical user interface of a client device.

2 . The method of claim 1 , wherein the recommending a candidate product further comprises ranking the candidate product in a product list based on the visual similarity measure.

3 . The method of claim 2 , wherein the candidate product has a highest ranking in the candidate product list.

4 . The method of claim 3 , wherein the causing presentation of the recommended candidate product comprises presenting the candidate product at a top position of the candidate product list.

5 . The method of claim 1 , wherein the candidate product image is associated with an electronic marketplace.

6 . The method of claim 1 , wherein the machine learning model comprises a neural network.

7 . The method of claim 6 , further comprising retraining the neural network.

8 . A non-transitory computer-readable storage medium having embedded therein a set of instructions which, when executed by one or more processors of a computer, causes the computer to execute operations comprising:
receiving an input query image; analyzing the input query image using a machine learning model to identify input query image visual text content; determining a visual similarity measure between a candidate product image visual text content and the input query image visual text content based on image signatures associated with the candidate product image and the input query image; recommending a candidate product based on the visual similarity measure; and causing presentation of the recommended candidate product on a graphical user interface of a client device.

9 . The non-transitory computer-readable storage medium of claim 8 , wherein the recommending a candidate product further comprises ranking the candidate product in a product list based on the visual similarity measure.

10 . The non-transitory computer-readable storage medium of claim 9 , wherein the candidate product has a highest ranking in the candidate product list.

11 . The non-transitory computer-readable storage medium of claim 10 , wherein the causing presentation of the recommended candidate product comprises presenting the candidate product at a top position of the candidate product list.

12 . The non-transitory computer-readable storage medium of claim 8 , wherein the candidate product image is associated with an electronic marketplace.

13 . The non-transitory computer-readable storage medium of claim 8 , wherein the machine learning model comprises a neural network.

14 . The non-transitory computer-readable storage medium of claim 13 , wherein the operations further comprise retraining the neural network.

15 . A system comprising:
at least one processor; and memory encoding computer-executable instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
receiving an input query image;
analyzing the input query image using a machine learning model to identify input query image visual text content;
determining a visual similarity measure between a candidate product image visual text content and the input query image visual text content based on image signatures associated with the candidate product image and the input query image;
recommending a candidate product based on the visual similarity measure; and
causing presentation of the recommended candidate product on a graphical user interface of a client device.

16 . The system of claim 15 , wherein the recommending a candidate product further comprises ranking the candidate product in a product list based on the visual similarity measure.

17 . The system of claim 16 , wherein the candidate product has a highest ranking in the candidate product list.

18 . The system of claim 17 , wherein the causing presentation of the recommended candidate product comprises presenting the candidate product at a top position of the candidate product list.

19 . The system of claim 15 , wherein the candidate product image is associated with an electronic marketplace.

20 . The system of claim 15 , wherein the machine learning model comprises a neural network.

Assignees

• Ebay Inc

Inventors

• Zheng Shuai
• Piramuthu Robinson

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06T2207/30108

  Industrial image inspection · CPC title

• G06T7/0004

  Industrial image inspection · CPC title