Method and system for generating images using generative adversarial networks (GANs)

Patent metadata

Field	Value
Publication number	US-12243133-B2
Application number	US-202217734938-A
Country	US
Kind code	B2
Filing date	May 2, 2022
Priority date	May 2, 2022
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

An image processing method and system that generates output images. The system receives a first input image depicting a first set of products and determines the first set of products and corresponding first product categories. The system then receives, on a user interface of a requestor device, a second input image depicting other products selected as being of interest having corresponding second product categories for the other products. In response to a match between one of the first product categories and the second product categories: the system applies the first input image and the second input image to generative adversarial networks (GANs). Each GAN is trained using image dataset for corresponding ones of the first and second product categories, to generate an output image replacing at least a portion of first input image with the second input image, the replacement based on the match between the product categories.

First claim

The invention claimed is:

1. A computer implemented method comprising:
receiving a first input image depicting a first set of products;
determining from the first input image, the first set of products and corresponding first product categories;
receiving, via a user interface of a requestor device, a second input image depicting other products having corresponding second product categories, the second input image selected as being of interest; and,
matching the first product categories with the second product categories and using one or more generative adversarial networks (GANs) to generate an output image based on one of the first product categories matching the second product categories, the generating including applying the first input image and the second input image to the one or more GANs trained using image dataset for corresponding ones of the first and second product categories, to generate the output image replacing at least a portion of first input image with the second input image, the replacement based on the matching between one of the first product categories and the second product categories.

2. The method of claim 1 , further comprising: providing a classifier for determining based on the first input image, the first set of products and for determining the second product categories for the other products.

3. The method of claim 2 , wherein the classifier was trained based on a dataset of images to detect product categories.

4. The method of claim 3 , wherein the first input image comprises the first set of products depicted in use on at least one of an object or a person.

5. The method of claim 4 further comprising:
detecting profile data for a user of the requestor device selecting products of interest;
accessing a database of mapping of the profile data to a set of visual attributes for the user; and,
modifying the person depicted in the first input image and thereby the output image based on the set of visual attributes.

6. The method of claim 3 , wherein the first or the second input image is selected and received in response to interactions with an e-commerce website, the interactions comprising one or more of:
browsing an e-commerce product on the e-commerce website;
selecting an e-commerce product on the e-commerce website for detailed viewing; or
adding an e-commerce product to an electronic cart of the e-commerce website.

7. The method of claim 6 , wherein the second input image is further selected based on a recommender model detecting a browsing history of the interactions on the e-commerce website, the browsing history used by the recommender model to detect additional products of interest based on the recommender model being trained on prior browsing history for a set of requestor devices associated with the additional products of interest.

8. The method of claim 3 , wherein the classifier was trained on the dataset of images comprising image features associated with each of the images; one or more labelled product categories for each of the images; and a boundary box visually defined around each of the labelled product categories in each image.

9. The method of claim 3 , wherein the classifier is a convolutional neural network classifier.

10. The method of claim 3 , wherein the first and the second input images depict one or more e-commerce products, the e-commerce products comprising articles of clothing.

11. The method of claim 3 , wherein the GANs are trained, using the second image dataset comprises images showing at least one of: person images or product images showing at least e-commerce products.

12. The method of claim 3 , further comprising providing a machine learning model trained using historical product and person image data of actual products and persons, the machine learning model receiving as input the output image and classifying the output image as realistic or not based on the output image of the model exceeding a defined confidence score.

13. The method of claim 3 , further comprising: determining a priority generation value for the first product categories and the second product categories, and wherein generating the output image replacing said portion of the first input image with the second input image, the replacement occurring only when the priority generation value of corresponding second product categories exceeds the first product categories.

14. The method of claim 3 , wherein the GANs are configured to replace the portion of the first input image with the second input image, only when a resulting combination of products in a potential output image satisfies a matching trigger.

15. The method of claim 1 , wherein the first input image is first generated by a generative adversarial network (GAN).

16. The method of claim 15 , wherein the generating of the first input image and the generating of the output image are performed by a same generative adversarial network.

17. A non-transitory computer readable medium having instructions tangibly stored thereon configured for generating output images, wherein the instructions, when executed cause a system to:
receive a first input image depicting a first set of products;
determine from the first input image, the first set of products and corresponding first product categories;
receive, via a user interface of a requestor device, a second input image depicting other products having corresponding second product categories, the second input image selected as being of interest; and,
match the first product categories with the second product categories and using one or more generative adversarial networks (GANs) to generate an output image based on one of the first product categories matching the second product categories, the generation including applying the first input image and the second input image to the one or more GANs trained using image dataset for corresponding ones of the first and second product categories, to generate the output image replacing at least a portion of first input image with the second input image, the replacement based on the matching between one of the first product categories and the second product categories.

18. A computer system for generating output images, the computer system comprising:
a processor in communication with a storage, the processor configured to execute instructions stored on the storage to cause the system to:
receive a first input image depicting a first set of products;
determine from the first input image, the first set of products and corresponding first product categories;
receive, via a user interface of a requestor device, a second input image depicting other products having corresponding second product categories, the second input image selected as being of interest; and,
match the first product categories with the second product categories and using one or more generative adversarial networks (GANs) to generate an output image based on one of the first product categories matching the second product categories, the generation including applying the first input image and the second input image to the one or more GANs trained using image dataset for corresponding ones of the first and second product categories, to generate the output image replacing at least a portion of first input image with the second input image, the replacement based on the matching between one of the first product categories and the second product categories.

19. The system of claim 18 , further comprising: a classifier configured for determining on the first input image, the first set of product

Assignees

• Shopify Inc

Inventors

• Wright Asher
• Jayatunga Hettige Ray Perera

Classifications

• G06Q30/0643

  graphically representing goods, e.g. 3D product representation · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06Q30/0633 Primary

  Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087) · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06V10/82

  using neural networks · CPC title