Deep learning based visual compatibility prediction for bundle recommendations

Patent metadata

Field	Value
Publication number	US-12136118-B2
Application number	US-202318186528-A
Country	US
Kind code	B2
Filing date	Mar 20, 2023
Priority date	May 4, 2020
Publication date	Nov 5, 2024
Grant date	Nov 5, 2024

Abstract

Systems, methods, and computer storage media are disclosed for predicting visual compatibility between a bundle of catalog items (e.g., a partial outfit) and a candidate catalog item to add to the bundle. Visual compatibility prediction may be jointly conditioned on item type, context, and style by determining a first compatibility score jointly conditioned on type (e.g., category) and context, determining a second compatibility score conditioned on outfit style, and combining the first and second compatibility scores into a unified visual compatibility score. A unified visual compatibility score may be determined for each of a plurality of candidate items, and the candidate item with the highest unified visual compatibility score may be selected to add to the bundle (e.g., fill the in blank for the partial outfit).

First claim

What is claimed is:

1. One or more non-transitory computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
generating a compatibility score that quantifies visual compatibility between style of a partial outfit of items and style of a candidate item to add to the partial outfit based on feeding node embeddings, of items in the partial outfit generated by an encoder of a first neural network, into a second neural network comprising a style autoencoder to generate item style embeddings for the items and computing a style embedding for the partial outfit as a weighted combination of the item style embeddings, weighted based on pairwise similarities predicted by a decoder of the first neural network; and
performing one or more pre-determined actions based on the compatibility score.

2. The one or more non-transitory computer storage media of claim 1 , the operations further comprising using a learned transformation function to combine the compatibility score and another compatibility score into a unified visual compatibility score, wherein the performing of the one or more pre-determined actions is based on the unified visual compatibility score.

3. The one or more non-transitory computer storage media of claim 1 , wherein computing the style embedding for the partial outfit comprises weighting the item style embeddings by corresponding outfit style attentions computed using the predicted pairwise similarities.

4. The non-transitory one or more computer storage media of claim 1 , wherein generating the compatibility score further comprises:
generating, for each of the items in the partial outfit, an outfit style attention based on the predicted pairwise similarities predicted by the first neural network decoding the node embeddings of the items; and
generating, based on the feeding of the node embeddings of the items generated by the first neural network into the second neural network, the style embedding for the partial outfit weighted by the outfit style attention for each of the items in the partial outfit.

5. The one or more non-transitory computer storage media of claim 1 , wherein generating the compatibility score further comprises:
using the style autoencoder to predict a first outfit mixture ratio in a style basis corresponding to the partial outfit;
using the style autoencoder to predict a second outfit mixture ratio in the style basis corresponding to the candidate item to add to the partial outfit; and
generating the compatibility score based on a decrease in uncertainty from the first outfit mixture ratio to the second outfit mixture ratio.

6. The one or more non-transitory computer storage media of claim 1 , wherein the performing of the one or more pre-determined actions is based on a unified visual compatibility generated by combining the compatibility score with another compatibility score using a transformation function, the operations further comprising learning the transformation function using a recurrent neural network controller configured to repetitively predict components of a core unit to form a composite function that combines and weights the compatibility score and the other compatibility score.

7. The one or more non-transitory computer storage media of claim 1 , wherein the performing of the one or more pre-determined actions is based on a unified visual compatibility generated by combining the compatibility score with another compatibility score, wherein the one or more pre-determined actions comprise causing presentation of the unified visual compatibility score or adding the candidate item to the partial outfit.

8. A computerized method comprising:
for each candidate item of a plurality of candidate items, of a catalog, to fill in a blank in a bundle of items of the catalog, generating a compatibility score quantifying compatibility between the candidate item and the bundle, at least in part by feeding node embeddings of the items generated by an encoder of a graph neural network into a second neural network comprising a style autoencoder to generate item style embeddings for the items and computing a style embedding for the bundle as a weighted combination of the item style embeddings, weighted based on pairwise similarities predicted by a decoder of the graph neural network; and
performing at least one of causing a presentation of a first candidate item having a highest compatibility score or adding the first candidate item to the bundle to fill in the blank.

9. The computerized method of claim 8 , further comprising using a learned transformation function to combine the compatibility score and another compatibility score into a unified visual compatibility score, wherein the highest compatibility score is a highest unified visual compatibility score.

10. The computerized method of claim 8 , wherein computing the style embedding for the bundle comprises weighting the item style embeddings by corresponding outfit style attentions computed using the predicted pairwise similarities.

11. The computerized method of claim 8 , wherein generating the compatibility score further comprises:
generating, for each of the items in the bundle of items, an outfit style attention based on the predicted pairwise similarities predicted by the graph neural network decoding the node embeddings of the items; and
generating, based on the feeding of the node embeddings of the items generated by the graph neural network into the second neural network, the style embedding for the bundle of items weighted by the outfit style attention for each of the items in the bundle of items.

12. The computerized method of claim 8 , wherein generating the compatibility score further comprises:
using the style autoencoder to predict a first outfit mixture ratio in a style basis corresponding to the bundle of items;
using the style autoencoder to predict a second outfit mixture ratio in the style basis corresponding to the candidate item to add to the bundle of items; and
generating the compatibility score based on a decrease in uncertainty from the first outfit mixture ratio to the second outfit mixture ratio.

13. The computerized method of claim 8 , further comprising using a transformation function to combine the compatibility score and another compatibility score into a unified visual compatibility score, the computerized method further comprising learning the transformation function using a recurrent neural network controller configured to repetitively predict components of a core unit to form a composite function that combines and weights the compatibility score and the other compatibility score.

14. A computer system comprising one or more processors and memory configured to provide computer program instructions to the one or more processors, the computer program instructions comprising:
an outfit style compatibility scoring component configured to direct the one or more processors to generate a compatibility score that quantifies compatibility between a partial outfit of items and a candidate item to add to the partial outfit based on feeding node embeddings of the items generated by an encoder of a first neural network into a second neural network comprising a style autoencoder and computing a style embedding for the partial outfit as a weighted combination of the item style embeddings, weighted based on pairwise similarities predicted by a decoder of the first neural network; and
a visual compatibility scoring component configured to direct the one or more processors to generate a recommendation to add the c

Assignees

• Adobe Inc

Inventors

• Ayush Kumar
• Chopra Ayush
• Govind Patel U
• Krishnamurthy Balaji
• Singhal Anirudh

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title