Intelligent online personal assistant with offline visual search database

Patent metadata

Field	Value
Publication number	US-11748978-B2
Application number	US-201615294767-A
Country	US
Kind code	B2
Filing date	Oct 16, 2016
Priority date	Oct 16, 2016
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

Systems, methods, and computer program products for identifying a candidate product in an electronic marketplace based on a visual comparison between candidate product image visual content and input query image visual content. Embodiments generate and store descriptive image signatures from candidate product images or selected portions of such images. A subsequently calculated visual similarity measure serves as a visual search result score for the candidate product in comparison to an input query image. Any number of images of any number of candidate products may be analyzed, such as for items available for sale in an online marketplace. Image analysis results are stored in a database and made available for subsequent automated on-demand visual comparisons to an input query image. The embodiments enable substantially real time visual based product searching of a potentially vast catalog of items.

First claim

What is claimed is:

1. A method comprising:
receiving an input query image from an interactive chat window on a client device;
generating, in real-time using a neural network, an image signature for the input query image, the image signature numerically describing image content;
receiving, using one or more processors, from a database, an offline generated image signature for each candidate product image in an electronic marketplace, the respective offline generated image signatures being previously generated based on an offline image signature generation process;
calculating, in real-time, a visual similarity measure between each candidate product image and the input query image based on the respective corresponding image signatures, the visual similarity measure indicating a correlation between each candidate product image and the input query image;
determining a ranked list of candidate products based on the visual similarity measure and knowledge graph data, the determining comprising:
generating, based on the correlation indicated by the visual similarity measure, the ranked list of candidate products; and
confirming, based on the knowledge graph data, the correlation of each candidate product image of the ranked list of candidate products generated based on the visual similarity measure; and

causing presentation of the ranked list of candidate products in a graphical user interface that includes the interactive chat window on the client device.

2. The method of claim 1 , wherein the generating the image signatures is performed for new products listed in the electronic marketplace on a periodic basis or an ongoing basis.

3. The method of claim 1 , wherein the image signature indicates a category of a product.

4. The method of claim 3 , wherein the image signature further indicates an aspect of the product.

5. The method of claim 3 , wherein the determining the ranked list of candidate products is further based on the category.

6. The method of claim 1 , wherein each candidate product has multiple associated candidate product images, each similarly processed.

7. The method of claim 1 , further comprising updating the database with the image signature for the input query image.

8. A non-transitory computer-readable storage medium having embedded therein a set of instructions which, when executed by one or more processors of a computer, causes the computer to perform operations comprising:
receiving an input query image from an interactive chat window on a client device;
generating, in real-time using a neural network, an image signature for the input query image, the image signature numerically describing image content;
receiving, from a database, an offline generated image signature for each candidate product image in an electronic marketplace, the respective offline generated image signatures being previously generated based on an offline image signature generation process;
calculating, in real-time, a visual similarity measure between each candidate product image and the input query image based on the respective corresponding image signatures, the visual similarity measure indicating a correlation between each candidate product image and the input query image;
determining, a ranked list of candidate products based on the visual similarity measure and knowledge graph data, the determining comprising:
generating, based on the correlation indicated by the visual similarity measure, the ranked list of candidate products; and
confirming based on the knowledge graph data, the correlation of each candidate product image of the ranked list of candidate products generated based on the visual similarity measure; and

causing presentation of the ranked list of candidate products in a graphical user interface that includes the interactive chat window on the client device.

9. The medium of claim 8 , wherein the generating the image signatures is performed for new products listed in the electronic marketplace on a periodic basis or an ongoing basis.

10. The medium of claim 8 , wherein the generating the image signatures is performed by a plurality of processors operating in parallel.

11. The medium of claim 8 , wherein the candidate product images correspond to products in a live inventory of the electronic marketplace.

12. The medium of claim 8 , wherein the input query image comprises a plurality of input query images, each similarly processed.

13. The medium of claim 8 , wherein each candidate product has multiple associated candidate product images, each similarly processed.

14. The medium of claim 8 , further comprising updating the database with the image signature for the input query image.

15. A system comprising:
a memory comprising instructions; and
one or more hardware-based computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
receiving an input query image from an interactive chat window on a client device;
generating, in real-time using a neural network, an image signature for the input query image, the image signature numerically describing image content;
receiving, using one or more processors, from a database, an offline generated image signature for each candidate product image in an electronic marketplace, the respective offline generated image signatures being previously generated based on an offline image signature generation process;
calculating, in real-time, a visual similarity measure between each candidate product image and the input query image based on the respective corresponding image signatures, the visual similarity measure indicating a correlation between each candidate product image and the input query image;
determining, a ranked list of candidate products based on the visual similarity measure and knowledge graph data, the determining comprising:
generating, based on the correlation indicated by the visual similarity measure, the ranked list of candidate products; and
confirming based on the knowledge graph data, the correlation of each candidate product image of the ranked list of candidate products generated based on the visual similarity measure; and

causing presentation of the ranked list of candidate products in a graphical user interface that includes the interactive chat window on the client device.

16. The system of claim 15 , wherein the generating the image signatures is performed for new products listed in the electronic marketplace on a periodic basis or an ongoing basis.

17. The system of claim 15 , wherein the generating the image signatures is performed by a plurality of processors operating in parallel.

18. The system of claim 15 , wherein the candidate product images correspond to products in a live inventory of the electronic marketplace.

19. The system of claim 15 , wherein the input query image comprises a plurality of input query images, each similarly processed.

20. The system of claim 15 , wherein each candidate product has multiple associated candidate product images, each similarly processed.

Assignees

• Ebay Inc

Inventors

• Kale Ajinkya Gorakhnath
• Yang Fan
• Wang Qiaosong
• Kiapour Mohammadhadi
• Piramuthu Robinson

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06Q30/0629 Primary

  by pre-processing results, e.g. ranking or ordering results · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F16/248

  Presentation of query results · CPC title

• G06F16/24578

  using ranking · CPC title