Visual aspect localization presentation

Patent metadata

Field	Value
Publication number	US-11775844-B2
Application number	US-202117165481-A
Country	US
Kind code	B2
Filing date	Feb 2, 2021
Priority date	Mar 22, 2017
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

Various embodiments use a neural network to analyze images for aspects that characterize the images, to present locations of those aspects on the images, and, additionally, to permit a user to interact with those locations on the images. For example, a user may interact with a visual cue over one of those locations to modify, refine, or filter the results of a visual search, performed on a publication corpus, that uses an input image (e.g., one captured using a mobile device) as a search query.

First claim

What is claimed is:

1. A method comprising:
providing, by a hardware processor and as an input to a machine-learning system, an input image comprising an object, the object associated with a product category of an online marketplace;
receiving, from the machine-learning system, aspect value data indicating a plurality of predicted aspect values corresponding to a plurality of aspects of the object, the aspect value data including, for each of the plurality of aspects, a set of locations on the input image that caused prediction of the plurality of predicted aspect values;
generating, by the hardware processor and based on the aspect value data, a visual cue over the input image for a particular location in the set of locations, the visual cue selectable by user interaction on a display;
presenting, by the hardware processor, a plurality of images, each of the plurality of images comprising another object of a plurality of other objects of a same product category as the object, each of the plurality of other objects representing a different aspect value; and
filtering, by the hardware processor and responsive to user selection of a particular image of the plurality of images, results of a search for product listings of the online marketplace based on the input image.

2. The method of claim 1 , wherein each of the plurality of other objects share with the object an aspect of the plurality of aspects of the object.

3. The method of claim 1 , wherein filtering results of the search comprises:
receiving, by the hardware processor, the user selection of the particular image;
identifying, by the hardware processor, additional objects having a same aspect value of a particular object in the particular image; and
presenting, by the hardware processor, additional images, each of the additional images comprising one of the additional objects having the same aspect value of the particular object.

4. The method of claim 3 , further comprising:
receiving an additional user selection of the visual cue, the visual cue associated with a first aspect value of the plurality of predicted aspect values corresponding to a first aspect of the plurality of aspects of the object; and
presenting a second set of images comprising a second set of objects having one or more different aspect values corresponding to the first aspect based at least in part on the additional user selection of the visual cue associated with the first aspect value.

5. The method of claim 1 , wherein the filtering comprises aspect-value based filtering in response to the user selection of the particular image.

6. The method of claim 1 , wherein the aspect value data includes a set of strength values corresponding to the set of locations, a particular strength value in the set of strength values representing a relevance level of the particular location to a respective aspect value.

7. The method of claim 1 , wherein the visual cue over the input image comprises a bounding shape presented as a visual overlay.

8. A system comprising:
a storage device storing instructions; and
a hardware processor configured by the instructions to perform operations comprising:
providing, as an input to a machine-learning system, an input image comprising an object, the object associated with a product category of an online marketplace;
receiving, from the machine-learning system, aspect value data indicating a plurality of predicted aspect values corresponding to a plurality of aspects of the object, the aspect value data including, for each of the plurality of aspects, a set of locations on the input image that caused prediction of the plurality of predicted aspect values;
generating, based on the aspect value data, a visual cue over the input image for a particular location in the set of locations, the visual cue selectable by user interaction on a display;
presenting a plurality of images, each of the plurality of images comprising another object of a plurality of other objects of a same product category as the object, each of the plurality of other objects representing a different aspect value; and
filtering, responsive to user selection of a particular image of the plurality of images, results of a search for product listings of the online marketplace based on the input image.

9. The system of claim 8 , wherein each of the plurality of other objects share with the object an aspect of the plurality of aspects of the object.

10. The system of claim 8 , wherein filtering results of the search comprises:
receiving the user selection of the particular image;
identifying additional objects having a same aspect value of a particular object in the particular image; and
presenting additional images, each of the additional images comprising one of the additional objects having the same aspect value of the particular object.

11. The system of claim 10 , wherein the additional images are selectable through user interaction.

12. The system of claim 8 , wherein the filtering comprises aspect-value based filtering in response to the user selection of the particular image.

13. The system of claim 8 , wherein the aspect value data includes a set of strength values corresponding to the set of locations, a particular strength value in the set of strength values representing a relevance level of the particular location to a respective aspect value.

14. The system of claim 8 , wherein the visual cue over the input image comprises a bounding shape presented as a visual overlay.

15. A non-transitory machine-readable storage medium storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations comprising:
providing, as an input to a machine-learning system, an input image comprising an object, the object associated with a product category of an online marketplace;
receiving, from the machine-learning system, aspect value data indicating a plurality of predicted aspect values corresponding to a plurality of aspects of the object, the aspect value data including, for each of the plurality of aspects, a set of locations on the input image that caused prediction of the plurality of predicted aspect values;
generating, based on the aspect value data, a visual cue over the input image for a particular location in the set of locations, the visual cue selectable by user interaction on a display;
presenting a plurality of images, each of the plurality of images comprising another object of a plurality of other objects of a same product category as the object, each of the plurality of other objects representing a different aspect value; and
filtering, responsive to user selection of a particular image of the plurality of images, results of a search for product listings of the online marketplace based on the input image.

16. The non-transitory machine-readable storage medium of claim 15 , wherein each of the plurality of other objects share with the object an aspect of the plurality of aspects of the object.

17. The non-transitory machine-readable storage medium of claim 15 , wherein filtering results of the search comprises:
receiving, by the hardware processor, the user selection of the particular image;
identifying, by the hardware processor, additional objects having a same aspect value of a particular object in the particular image; and
presenting, by the hardware processor, additional images, each of the additional images comprising one of the additional objects having the same aspect value of the particular object.

18. The non-transitory mach

Assignees

• Ebay Inc

Inventors

• Yu Licheng
• Kiapour Mohammadhadi
• Kale Ajinkya Gorakhnath
• Piramuthu Robinson

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title