Visual aspect localization presentation

Patent metadata

Field	Value
Publication number	US-10943176-B2
Application number	US-201715465883-A
Country	US
Kind code	B2
Filing date	Mar 22, 2017
Priority date	Mar 22, 2017
Publication date	Mar 9, 2021
Grant date	Mar 9, 2021

Abstract

Various embodiments use a neural network to analyze images for aspects that characterize the images, to present locations of those aspects on the images, and, additionally, to permit a user to interact with those locations on the images. For example, a user may interact with a visual cue over one of those locations to modify, refine, or filter the results of a visual search, performed on a publication corpus, that uses an input image (e.g., one captured using a mobile device) as a search query.

First claim

What is claimed is:

1. A method comprising:
providing, by a hardware processor, an input image as an input to a machine-learning system comprising a trained neural network;
receiving, from the machine-learning system, a first category prediction of an object detected in the input image;
receiving, from the machine-learning system, aspect value data indicating a plurality of predicted aspect values for a corresponding plurality of aspects of the object, each of the plurality of aspects specific to the predicted first category of the object and each of the plurality of predicted aspect values further characterizing the object of the predicted first category, the aspect value data further including, for each of the plurality of aspects, a set of locations on the input image that are associated with a corresponding one of the plurality of predicted aspect values; and
generating, by the hardware processor and based on the aspect value data, a visual cue over the input image for a particular location in the set of locations, the visual cue selectable by user interaction on a display.

2. The method of claim 1 , wherein the aspect value data includes a set of strength values corresponding to the set of locations, a particular strength value in the set of strength values representing a relevance level of the particular location to the aspect value.

3. The method of claim 2 , wherein the visual cue indicates the particular strength value.

4. The method of claim 2 , comprising generating, by the hardware processor, a filtered set of locations by filtering the set of locations based on a strength threshold and the set of strength values, the filtered set of locations including the particular location, and wherein the machine-learning system comprises a trained neural network, and the set of locations included by the aspect value data is determined based on information from of a convolution layer of the trained neural network.

5. The method of claim 1 , wherein the aspect value data includes a probability value that the object depicted in the input image has the aspect value, and the visual cue indicates the probability value.

6. The method of claim 1 , wherein the visual cue comprises at least one of a heat map-based visual cue, a call out, a point, a bounding shape, or a shading.

7. The method of claim 1 , comprising responsive to receiving a user interaction with respect to the visual cue, causing presentation of a set of images that corresponds to a set of other objects, each particular object in the set of other objects representing a different aspect value of the particular aspect.

8. The method of claim 7 , comprising responsive to a user selection of a particular image in the set of images that corresponds to a specific other object in the set of other objects, filtering, by the hardware processor, results of a visual search performed based on the input image, the specific other object representing a specific aspect value of the particular aspect, and the filtering being based on the specific aspect value.

9. The method of claim 1 , further comprising:
receiving, from the machine-learning system, a second category prediction of a second object detected in a second image, the second category prediction different than the first category prediction;
receiving, from the machine-learning system, second aspect value data indicating a second plurality of aspect values for a corresponding second plurality of aspects of the second object, the second plurality of aspects specific to the predicted second category and configured to further characterize objects of the predicted second category, the second plurality of aspects different than the plurality of aspects characterizing the object detected in the input image, the aspect value data further including a second set of locations on the second image that relate to an aspect value; and
generating, by the hardware processor and based on the aspect value data, a second visual cue over the second image for a second particular location in the second set of locations.

10. A computer comprising:
a storage device storing instructions; and
a hardware processor configured by the instructions to perform operations comprising:
providing an input image as an input to a machine-learning system comprising a trained neural network;
receiving, from the machine-learning system, a category prediction of an object detected in the input image;
receiving, from the machine-learning system, aspect value data indicating a plurality of predicted aspect values for a corresponding plurality of aspects of the object, each of the plurality of aspects specific to the predicted first category of the object and each of the plurality of aspect values further characterizing the object of the predicted first category, the aspect value data further including, for each of the plurality of aspects, a set of locations on the input image that are associated with a corresponding one of the plurality of predicted aspect values; and
generating, based on the aspect value data, a visual cue over the input image for a particular location in the set of locations, the visual cue selectable by user interaction on a display.

11. The computer of claim 10 , wherein the aspect value data includes a set of strength values corresponding to the set of locations, a particular strength value in the set of strength values representing a relevance level of the particular location to the aspect value.

12. The computer of claim 11 , wherein the visual cue indicates the particular strength value.

13. The computer of claim 11 , the operations further comprising generating a filtered set of locations by filtering the set of locations based on a strength threshold and the set of strength values, the filtered set of locations including the particular location.

14. The computer of claim 10 , wherein the aspect value data includes a probability value that the object depicted in the input image has the aspect value, and the visual cue indicates the probability value.

15. The computer of claim 10 , wherein the visual cue comprises at least one of a heat map-based visual cue, a call out, a point, a bounding shape, or a shading.

16. The computer of claim 10 , the operations further comprising responsive to receiving a first user interaction with respect to the visual cue, causing presentation of a set of images that corresponds to a set of other objects, each particular object in the set of other objects representing a different aspect value of the particular aspect.

17. The computer of claim 16 , wherein the hardware processor is configured by the instructions to perform operations comprising responsive to a user selection of a particular image in the set of images that corresponds to a specific other object in the set of other objects, filtering results of a visual search performed based on the input image, the specific other object representing a specific aspect value of the particular aspect, and the filtering being based on the specific aspect value.

18. A method comprising:
providing, by a hardware processor, an input image as an input to a machine-learning system comprising a trained neural network;
receiving, from the machine-learning system, a category prediction of an object detected in the input image;
receiving, from the machine-learning system, aspect value data indicating a plurality of predicted aspect values for a corresponding plurality of aspects of the object, each of the plurality of aspects specific to the predicted category of the object and each of the plurality of

Assignees

• Ebay Inc

Inventors

• Yu Licheng
• Kiapour Mohammadhadi
• Kale Ajinkya Gorakhnath
• Piramuthu Robinson

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/82

  using neural networks · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title