Image analysis and prediction based visual search

Patent metadata

Field	Value
Publication number	US-12050641-B2
Application number	US-202318228328-A
Country	US
Kind code	B2
Filing date	Jul 31, 2023
Priority date	Oct 16, 2016
Publication date	Jul 30, 2024
Grant date	Jul 30, 2024

Abstract

Methods, systems, and computer programs are presented for adding new features to a network service. A method includes receiving an image depicting an object of interest. A category set is determined for the object of interest and an image signature is generated for the image. Using the category set and the image signature, the method identifies a set of publications within a publication database and assigns a rank to each publication. The method causes presentation of the ranked list of publications at a computing device from which the image was received.

First claim

What is claimed is:

1. A method, comprising:
receiving a video comprising a set of frames from a user device;
generating, via one or more hardware processors, a composite image of two or more frames of the video;
determining, via the one or more hardware processors, a category set of the composite image;
identifying, based at least in part on the category set, a set of publications from a publications database; and
causing presentation, at the user device, of one or more publications from the set of publications.

2. The method of claim 1 , wherein the video is received, over a network, as part of a natural language communication exchange between the user device and a personal assistant system implemented by one or more server devices.

3. The method of claim 2 , wherein the one or more publications are transmitted, by the personal assistant system and over the network, as part of the natural language communication exchange and as a response to the video received from the user device.

4. The method of claim 1 , wherein determining the category set includes:
identifying an object of interest depicted in the composite image; and
selecting, from the publications database and based on the object of interest, one or more publication categories for inclusion in the category set.

5. The method of claim 1 , wherein the publications database includes a plurality of publication categories each associated with an image set having one or more representative images, and determining the category set includes:
generating a target semantic vector representing the composite image;
accessing source semantic vectors associated with respective image sets of respective publication categories; and
identifying one or more of the source semantic vectors that are a closest match to the target semantic vector, the category set including the publication categories corresponding to the identified one or more source semantic vectors.

6. The method of claim 1 , wherein the publications database includes a plurality of publication categories each associated with a keyword set having one or more keywords, and determining the category set includes:
identifying visual aspects of the composite image;
determining descriptive words corresponding to the visual aspects;
determining matching keywords in respective keyword sets of respective publication categories that match the descriptive words; and
identifying one or more of the keyword sets having a number or percentage of the matching keywords that exceeds a threshold, the category set including the publication categories associated with the identified one or more keyword sets.

7. The method of claim 1 , wherein identifying the set of publications includes:
identifying one or more publication categories of the publications database corresponding to the category set;
selecting query publications from the publications database associated with the one or more publication categories;
generating an image signature of the composite image; and
identifying the set of publications as a subset of the query publications based on the image signature.

8. The method of claim 7 , wherein identifying the set of publications includes:
accessing publication image signatures of the query publications;
assigning a rank to each of the query publications by comparing the image signature to the publication image signatures; and
selecting, as the set of publications, the subset of the query publications based on the rank.

9. The method of claim 8 , wherein assigning the rank includes determining distances between the image signature and the publication image signatures using a distance function, and selecting the subset of the query publications includes selecting the subset of the query publications having the distances that are below a threshold.

10. The method of claim 8 , wherein assigning the rank includes:
identifying visual aspects of the composite image;
determining descriptive words corresponding to the visual aspects;
accessing keywords associated with the query publications; and
assigning the rank to each of the query publications by comparing the descriptive words to the keywords.

11. The method of claim 8 , wherein causing presentation of the one or more publications includes generating a ranked list based on the rank assigned to individual publications in the set of publications, the one or more publications presented in a user interface of the user device based on the ranked list.

12. A system, comprising:
at least one processor; and
at least one memory including instructions which, when executed by the at least one processor, cause the system to perform operations comprising:
receiving a video comprising a set of frames from a user device;
generating a composite image of two or more frames of the video;
determining a category set of the composite image;
identifying, based at least in part on the category set, a set of publications from a publications database; and
causing presentation, at the user device, of one or more publications from the set of publications.

13. The system of claim 12 , wherein the video is received, over a network, as part of a natural language communication exchange between the user device and a personal assistant system implemented by one or more server devices.

14. The system of claim 13 , wherein the one or more publications are transmitted, by the personal assistant system and over the network, as part of the natural language communication exchange and as a response to the video received from the user device.

15. The system of claim 12 , wherein the publications database includes a plurality of publication categories each associated with an image set having one or more representative images, and determining the category set includes:
generating a target semantic vector representing the composite image;
accessing source semantic vectors associated with respective image sets of respective publication categories; and
identifying one or more of the source semantic vectors that are a closest match to the target semantic vector, the category set including the publication categories corresponding to the one or more source semantic vectors.

16. The system of claim 12 , wherein identifying the set of publications includes:
identifying one or more publication categories of the publications database corresponding to the category set;
selecting query publications from the publications database associated with the one or more publication categories;
generating an image signature of the composite image; and
identifying the set of publications as a subset of the query publications based on the image signature.

17. The system of claim 16 , wherein identifying the set of publications includes:
accessing publication image signatures of the query publications;
assigning a rank to each of the query publications by comparing the image signature to the publication image signatures; and
selecting, as the set of publications, the subset of the query publications based on the rank.

18. The system of claim 17 , wherein assigning the rank includes determining distances between the image signature and the publication image signatures using a distance function, and selecting the subset of the query publications includes selecting the subset of the query publications having the distances that are below a threshold.

19. The system of claim 17 , wherein causing presentation of the one or more publications includes generating a ranked list based on the rank assigned to in

Assignees

• Ebay Inc

Inventors

• Yang Fan
• Kale Ajinkya Gorakhnath
• Wang Qiaosong
• Kiapour Mohammadhadi
• Piramuthu Robinson

Classifications

• G06F16/55

  Clustering; Classification · CPC title

• G06F16/53

  Querying · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title