Deep learning based visual compatibility prediction for bundle recommendations

Patent metadata

Field	Value
Publication number	US-11640634-B2
Application number	US-202016865572-A
Country	US
Kind code	B2
Filing date	May 4, 2020
Priority date	May 4, 2020
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

Systems, methods, and computer storage media are disclosed for predicting visual compatibility between a bundle of catalog items (e.g., a partial outfit) and a candidate catalog item to add to the bundle. Visual compatibility prediction may be jointly conditioned on item type, context, and style by determining a first compatibility score jointly conditioned on type (e.g., category) and context, determining a second compatibility score conditioned on outfit style, and combining the first and second compatibility scores into a unified visual compatibility score. A unified visual compatibility score may be determined for each of a plurality of candidate items, and the candidate item with the highest unified visual compatibility score may be selected to add to the bundle (e.g., fill the in blank for the partial outfit).

First claim

What is claimed is:

1. One or more non-transitory computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
generating a first compatibility score, that quantifies compatibility between a partial outfit of items and a candidate item to add to the partial outfit and is jointly conditioned on item category and item context, based on feeding a representation of category co-occurrence and an incomplete graph representing the items in the partial outfit as connected nodes into a first neural network comprising an autoencoder with a graph neural network;
generating a second compatibility score that quantifies visual compatibility between style of the partial outfit and style of the candidate item based on feeding node embeddings of the items generated by the first neural network into a second neural network comprising a style autoencoder;
using a learned transformation function to combine the first compatibility score and the second compatibility score into a unified visual compatibility score; and
performing one or more pre-determined actions based on the unified visual compatibility score.

2. The one or more non-transitory computer storage media of claim 1 , wherein generating the first compatibility score further comprises:
representing catalog items, including the items in the partial outfit and the candidate item, as nodes of the incomplete graph with edges connecting nodes corresponding to the items in the partial outfit; and
using the autoencoder with the graph neural network to predict probabilities of missing edges in the incomplete graph.

3. The one or more non-transitory computer storage media of claim 1 , wherein generating the first compatibility score further comprises:
using the autoencoder with the graph neural network to predict pairwise similarities quantifying similarity between pairs of catalog items, the catalog items including the items in the partial outfit and the candidate item; and
generating the first compatibility score by averaging a set of the pairwise similarities between the candidate item and each of the items from the partial outfit.

4. The one or more non-transitory computer storage media of claim 1 , wherein generating the first compatibility score further comprises:
representing items of a catalog, including the items in the partial outfit and the candidate item, as the incomplete graph; and
using the autoencoder with the graph neural network to predict missing edges in the incomplete graph based on a corresponding adjacency matrix that is weighted by co-occurrence, within the catalog, of categories of items corresponding to pairs of the connected nodes.

5. The one or more non-transitory computer storage media of claim 1 , wherein generating the second compatibility score is further based at least in part on a style embedding for the partial outfit generated by attending over style embeddings for each of the items in the partial outfit.

6. The non-transitory one or more computer storage media of claim 1 , wherein generating the second compatibility score further comprises:
generating, for each of the items in the partial outfit, an outfit style attention based on pairwise similarities predicted by the first neural network decoding the node embeddings of the items; and
generating, based on feeding the node embeddings of the items generated by the first neural network into the second neural network, a style embedding for the partial outfit weighted by the outfit style attention for each of the items in the partial outfit.

7. The one or more non-transitory computer storage media of claim 1 , wherein generating the second compatibility score further comprises:
using the style autoencoder to predict a first outfit mixture ratio in a style basis corresponding to the partial outfit;
using the style autoencoder to predict a second outfit mixture ratio in the style basis corresponding to the candidate item being added to the partial outfit; and
generating the second compatibility score based on a decrease in uncertainty from the first outfit mixture ratio to the second outfit mixture ratio.

8. The one or more non-transitory computer storage media of claim 1 , the operations further comprising learning the transformation function using a recurrent neural network controller configured to repetitively predict components of a core unit to form a composite function that combines and weights the first compatibility score and the second compatibility score.

9. The one or more non-transitory computer storage media of claim 1 , wherein the one or more pre-determined actions comprise causing a presentation of the unified visual compatibility score or adding the candidate item to the partial outfit.

10. A computerized method comprising:
for each of a plurality of candidate items, of a catalog, to fill in a blank in a bundle of items of the catalog, generating a compatibility score quantifying compatibility between the candidate item and the bundle, at least in part by:
representing the items in the bundle as connected nodes in an incomplete graph;
processing the incomplete graph and a corresponding adjacency matrix that is weighted by co-occurrence, within the catalog, of categories of items corresponding to pairs of the connected nodes using a graph neural network to predict missing edges in the incomplete graph; and
feeding node embeddings of the items generated by the graph neural network into a second neural network comprising a style autoencoder; and

performing at least one of causing a presentation of a first candidate item having a highest compatibility score or adding the first candidate item to the bundle.

11. The computerized method of claim 10 ,
wherein the autoencoder is configured to predict pairwise similarities quantifying similarity between pairs of items in the catalog, and
wherein generating the compatibility score quantifying compatibility between the candidate item and the bundle further comprises generating a first compatibility score for the candidate item by averaging a set of the pairwise similarities between the candidate item and each of the items in the bundle.

12. The computerized method of claim 10 , wherein generating the compatibility score quantifying compatibility between the candidate item and the bundle is further based at least in part on a style embedding for the bundle generated by attending over style embeddings for each of the items in bundle.

13. The computerized method of claim 10 , wherein generating the compatibility score quantifying compatibility between the candidate item and the bundle further comprises:
generating, for each of the items in the bundle, a bundle style attention based on pairwise similarities predicted by a first neural network comprising the autoencoder with the graph neural network decoding the node embeddings of the items; and
generating, based on feeding the node embeddings of the items generated by the first neural network into the second neural network comprising a style autoencoder, a style embedding for the bundle weighted by the outfit style attention for each of the items in the bundle.

14. The computerized method of claim 10 , wherein generating the compatibility score quantifying compatibility between the candidate item and the bundle further comprises:
using a style autoencoder, distinct from the autoencoder with the graph neural network, to predict a first outfit mixture ratio in a style basis corresponding to the bundle;
using the style autoencoder to predict a second outfit mixture ratio in the

Assignees

• Adobe Inc

Inventors

• Ayush Kumar
• Chopra Ayush
• Govind Patel Utkarsh
• Krishnamurthy Balaji
• Singhal Anirudh

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title