Visual attribute determination for content selection

Patent metadata

Field	Value
Publication number	US-10846327-B2
Application number	US-201816179708-A
Country	US
Kind code	B2
Filing date	Nov 2, 2018
Priority date	Nov 2, 2018
Publication date	Nov 24, 2020
Grant date	Nov 24, 2020

Abstract

Content can be located for items that are stylistically similar to an item of interest. The item of interest can be represented in a query image, which is analyzed to determine one or more regions having an item represented therein. The classification of the item is determined, enabling identification of a trained model to be used to process image data for the region(s) of the query image. The trained model outputs a set of attributes, relating to visual or stylistic attributes, and corresponding confidence or prominence values for the attributes. These attributes and values can be compared against a data repository to locate items determined to be similar based on corresponding attributes and values. A similarity determination algorithm can identify similar items and rank those items by similarity. Content for the most similar items is returned as a result for the query image.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a query image including a representation of an item of interest;
locating the representation of the item of interest in the query image;
determining an item type of the item or interest;
processing the representation of the item using a trained machine learning model corresponding to the item type;
obtaining, from the trained machine learning model, a set of attributes and confidence values, the attributes including stylistics attributes exhibited by the representation of the item in the query image;
determining, using the set of attributes and confidence values, similarity scores for a set of similar items to the item of interest;
determining a ranking of the set of similar items, the ranking based at least in part on the similarity scores; and
providing content corresponding to at least a subset of the similar items, the subset based at least in part on the ranking.

2. The computer-implemented method of claim 1 , further comprising:
processing the query image using a localizer algorithm to determine a region of the query image including the representation of the item of interest.

3. The computer-implemented method of claim 2 , further comprising:
processing image data for the region using a trained classifier to determine the item type.

4. The computer-implemented method of claim 1 , wherein the stylistic attributes include at least one of a color, a pattern, a cut, a length, a shape, a silhouette, a neckline, a hemline, or an occasion type of the item of interest.

5. The computer-implemented method of claim 1 , further comprising:
training the trained machine learning model using a set of annotated images including items of the item type, wherein similarity of the items to the item of interest are able to be determined using a similarity determination algorithm accepting as input the attributes and confidence values.

6. A computer-implemented method, comprising:
processing an image using a trained model to produce a set of attributes representative of an item represented in the image, the attributes relating to at least one of visual attributes or stylistic attributes;
determining weighted relationships among the set of attributes for the item;
comparing the weighted relationships of the attributes against attribute data stored for items having been previously processed to identify a set of stylistically similar items having similar weighted relationships of attributes;
determining respective similarity scores for the set of stylistically similar items with respect to the item, the respective similarity scores based at least in part on the set of attributes;
ranking the stylistically similar items by the respective similarity scores;
determining a subset of the stylistically similar items based in part upon highest ranking by the respective similarity scores; and
providing content associated with at least the subset of the stylistically similar items.

7. The computer-implemented method of claim 6 , further comprising:
processing the image using a localizer algorithm to determine a region of the query image including the representation of the item.

8. The computer-implemented method of claim 7 , further comprising:
processing image data for the region using a trained classifier to determine an item type for the item.

9. The computer-implemented method of claim 8 , further comprising:
determining the weighted relationships by processing the representation of the item using a trained machine learning model corresponding to the item type.

10. The computer-implemented method of claim 6 , further comprising:
training the trained machine learning model using a set of annotated images including items of the item type, wherein similarity of the items to the item of interest are able to be determined using a similarity determination algorithm accepting as input the attributes and confidence values.

11. The computer-implemented method of claim 6 , wherein the stylistic attributes include at least one of a color, a pattern, a cut, a length, a shape, a silhouette, a neckline, a hemline, or an occasion type of the item of interest.

12. A system, comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to:
process an image using a trained model to produce a set of attributes representative of an item represented in the image, the attributes relating to at least one of visual attributes or stylistic attributes of the item;
determine weighted relationships among the set of attributes for the item;
compare the weighted relationships of the attributes against attribute data stored for items having been previously processed to identify a set of stylistically similar items having similar weighted relationships of attributes;
determine respective similarity scores for the set of stylistically similar items with respect to the item, the respective similarity scores based at least in part on the set of attributes;
rank the stylistically similar items by the respective similarity scores;
determine a subset of the stylistically similar items based in part upon highest ranking by the respective similarity scores; and
provide content associated with at least a subset of the stylistically similar items.

13. The system of claim 12 , wherein the instructions when executed further cause the system to:
process the image using a localizer algorithm to determine a region of the query image including the representation of the item.

14. The system of claim 13 , wherein the instructions when executed further cause the system to:
process image data for the region using a trained classifier to determine an item type for the item.

15. The computer-implemented method of claim 14 , further comprising:
determine the weighted relationships by processing the representation of the item using a trained machine learning model corresponding to the item type.

16. The system of claim 12 , wherein the instructions when executed further cause the system to:
train the trained machine learning model using a set of annotated images including items of the item type, wherein similarity of the items to the item of interest are able to be determined using a similarity determination algorithm accepting as input the attributes and confidence values.

Assignees

• A9 Com Inc

Inventors

• Salokhe Abhimanyu
• Guo Minghua
• Lai Eusebius

Classifications

• G06F16/5854 Primary

  using shape and object relationship · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title

• G06F16/532 Primary

  Query formulation, e.g. graphical querying · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06N20/00

  Machine learning · CPC title