Generating user-customized items using a visually-aware image generation network

Patent metadata

Field	Value
Publication number	US-10970765-B2
Application number	US-201815897856-A
Country	US
Kind code	B2
Filing date	Feb 15, 2018
Priority date	Feb 15, 2018
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

The present disclosure relates to a personalized fashion generation system that synthesizes user-customized images using deep learning techniques based on visually-aware user preferences. In particular, the personalized fashion generation system employs an image generative adversarial neural network and a personalized preference network to synthesize new fashion items that are individually customized for a user. Additionally, the personalized fashion generation system can modify existing fashion items to tailor the fashion items to a user's tastes and preferences.

First claim

What is claimed is:

1. A system for synthesizing user-customized images based on latent user preferences comprising:
a memory comprising:
an image generative adversarial network trained to generate realistic images of items; and
a preference predictor network trained to determine visual preferences of individual users;

at least one processor; and
at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:
generate a first realistic synthesized image utilizing the image generative adversarial network based on a latent code identified from a plurality of latent codes;
determine a first preference prediction score for the first realistic synthesized image utilizing the preference predictor network based on visual latent user features of a user;
generate a second realistic synthesized image utilizing the image generative adversarial network that is customized for the user based on a modified latent code modified according to the first preference prediction score, wherein the preference predictor network generates a second preference prediction score for the second realistic synthesized image that is greater than the first preference prediction score; and
provide the second realistic synthesized image customized for the user to a client device associated with the user.

2. The system of claim 1 , wherein:
the image generative adversarial network is trained utilizing a corpus of images of fashion items corresponding to fashion categories to identify latent representations of visual fashion characteristics; and
the realistic synthesized second image customized for the user comprises a synthesized image of a new fashion item generated for the user.

3. The system of claim 2 , further comprising instructions that, when executed by the at least one processor, cause the system to identify the plurality of latent codes within random latent space of the image generative adversarial network.

4. The system of claim 2 , wherein the image generative adversarial network comprises:
a generator neural network trained on the corpus of fashion images to generate synthesized images of fashion items; and
a discriminator neural network trained on the corpus of fashion images to determine when generated synthesized images of new fashion items resemble real images of fashion items.

5. The system of claim 4 , wherein
the image generative adversarial network trains in an unsupervised manner using the corpus of fashion images;
the image generative adversarial network alternates training the generator neural network and the discriminator neural network using objective functions via back propagation and least squares loss; and
the trained generator neural network generates synthesized images of fashion items following a same distribution of images from the corpus of fashion items.

6. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to determine the modified latent code from the plurality of latent codes based on utilizing a hyper-parameter that controls a trade-off between preference prediction scores and image quality.

7. The system of claim 1 , further comprising utilizing the preference predictor network to determine the first preference prediction score for the first realistic synthesized image based on correlating visual latent user features of the user with visual latent item features generated from the first realistic synthesized image by the image generative adversarial network.

8. The system of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the system to determine the modified latent code based on iteratively searching for a low-dimensional latent code within the plurality of latent codes that maximizes the preference prediction score for the user at the preference predictor network.

9. The system of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the system to determine the modified latent code by utilizing gradient ascent within a constrained space to identify an optimized latent code that yields a higher preference prediction score from the preference predictor network than the latent code.

10. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to repeat identifying the modified latent code by:
randomizing initial positions of latent codes within random latent space of the image generative adversarial network for a predetermined number of iterations;
based on the visual latent user features of the user, identify one or more optimized latent codes within the random latent space that yield higher preference prediction score for the user than the first preference prediction score; and
selecting the modified latent code from the one or more optimized latent codes based on the higher preference prediction scores for the user.

11. The system of claim 1 , wherein the plurality of latent codes comprises latent codes combined with random noise vectors within random latent space of the image generative adversarial network.

12. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generate a plurality of realistic synthesized images customized for the user; and
provide the plurality of realistic synthesized images customized for the user based on employing a probabilistic selection algorithm to increase diversity among the provided plurality of realistic synthesized images customized for the user.

13. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computer system to:
generate a first realistic synthesized image at an image generative adversarial network utilizing a latent code identified from a plurality of latent codes;
utilize a preference predictor network to determine a first preference prediction score for the first realistic synthesized image based on correlating visual latent user features of a user with visual latent item features generated from the first realistic synthesized image;
modify the latent code at the image generative adversarial network based on the first preference prediction score;
generate a second realistic synthesized image customized for the user at the image generative adversarial network utilizing the modified latent code, wherein the preference predictor network generates a second preference prediction score for the second realistic synthesized image that is greater than the first preference prediction score; and
provide the second realistic synthesized image customized for the user to a client device associated with the user.

14. The non-transitory computer-readable of claim 13 , further comprising instructions that, when executed by at least one processor, cause a computer system to modify the latent code by iteratively searching for one or more latent codes within random latent space of the image generative adversarial network that maximize preference a predictor score for the user at the preference predictor network.

15. The non-transitory computer-readable of claim 13 , wherein the image generative adversarial network comprises:
a generator neural network trained on a corpus of fashion images to generate synthesized images of fashion items; and
a discriminator neural network trained on the corpus of fashion images to determine when generated synthesized images of new fashion items resemble real images

Assignees

• Adobe Inc
• Univ California

Inventors

• Fang Chen
• Wang Zhaowen
• Kang Wangcheng
• Mcauley Julian

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title