Generating a digital image using a generative adversarial network

Patent metadata

Field	Value
Publication number	US-11869057-B2
Application number	US-202117539558-A
Country	US
Kind code	B2
Filing date	Dec 1, 2021
Priority date	Mar 16, 2018
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

Various embodiments described herein utilize multiple levels of generative adversarial networks (GANs) to facilitate generation of digital images based on user-provided images. Some embodiments comprise a first generative adversarial network (GAN) and a second GAN coupled to the first GAN, where the first GAN includes an image generator and at least two discriminators, and the second GAN includes an image generator and at least one discriminator. According to some embodiments, the (first) image generator of the first GAN is trained by processing a user-provided image using the first GAN. For some embodiments, the user-provided image and the first generated image, generated by processing the user-provided image using the first GAN, are combined to produce a combined image. For some embodiments, the (second) image generator of the second GAN is trained by processing the combined image using the second GAN.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
a computer-readable medium having instructions stored thereon, which, when executed by the one or more processors, cause the system to perform operations comprising:
receiving, from a client device, a user-provided image comprising an object of interest;
pre-processing the user-provided image to produce a pre-processed image;
training a first image generator by processing the pre-processed image using a first generative adversarial network;
generating, by the first image generator, based on the pre-processed image, a first generated image;
combining the pre-processed image and the first generated image to produce a combined image;
training a second image generator by processing the combined image using a second generative adversarial network;
generating, by the second image generator, based on the combined image, a second generated image;
performing, based on the first generated image or the second generated image, an image-based search to identify a set of one or more search result images, wherein each search result image of the set of one or more search result images comprises an object similar to the object of interest; and
providing, to the client device, the set of one or more search result images.

2. The system of claim 1 , wherein the user-provided image is associated with an item listing for the object similar to the object of interest on an online marketplace.

3. The system of claim 1 , wherein the user-provided image is a user-captured image.

4. The system of claim 1 , wherein a first search result image of the set of one or more search result images is associated with an item listing for the object similar to the object of interest on an online marketplace.

5. A method comprising:
receiving, from a client device, a user-provided image comprising an object of interest;
pre-processing the user-provided image to produce a pre-processed image;
training a first image generator by processing the pre-processed image using a first generative adversarial network;
generating, by the first image generator, based on the pre-processed image, a first generated image;
combining the pre-processed image and the first generated image to produce a combined image;
training a second image generator by processing the combined image using a second generative adversarial network;
generating, by the second image generator, based on the combined image, a second generated image;
performing, based on the first generated image or the second generated image, an image-based search to identify a set of one or more search result images, wherein each search result image of the set of one or more search result images comprises an object similar to the object of interest; and
providing, to the client device, the set of one or more search result images.

6. The method of claim 5 , wherein a first search result image of the set of one or more search result images is a stock-quality image.

7. The method of claim 5 , wherein the user-provided image is associated with an item listing for the object similar to the object of interest on an online marketplace.

8. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a system to perform operations comprising:
receiving, from a client device, a user-provided image comprising an object of interest;
pre-processing the user-provided image to produce a pre-processed image;
training a first image generator by processing the pre-processed image using a first generative adversarial network;
generating, by the first image generator, based on the pre-processed image, a first generated image;
combining the pre-processed image and the first generated image to produce a combined image;
training a second image generator by processing the combined image using a second generative adversarial network;
generating, by the second image generator, based on the combined image, a second generated image;
performing, based on the first generated image or the second generated image an image-based search to identify a set of one or more search result images, wherein each search result image of the set of one or more search result images comprises an object similar to the object of interest; and
providing, to the client device, the set of one or more search result images.

9. The non-transitory computer-readable medium of claim 8 , wherein the user-provided image is associated with an item listing for the object similar to the object of interest on an online marketplace.

10. The non-transitory computer-readable medium of claim 8 , wherein the user-provided image is a user-captured image.

11. The non-transitory computer-readable medium of claim 8 , wherein a first search result image of the set of one or more search result images is associated with an item listing for the object similar to the object of interest on an online marketplace.

12. The system of claim 1 , wherein a first search result image of the set of one or more search result images is a stock-quality image.

13. The system of claim 1 , wherein pre-processing the user-provided image comprises:
performing a person segmentation process on the user-provided image to produce a segmented image;
disposing one or more image segments of the segmented image on a background to produce an intermediate image;
performing a human pose estimation process on the intermediate image to identify a plurality of points of interest in the intermediate image, wherein the plurality of points of interest is associated with the object of interest; and
cropping the intermediate image based on the plurality of points of interest to produce the pre-processed image.

14. The system of claim 13 , wherein the background is a white background.

15. The method of claim 5 , wherein a first search result image of the set of one or more search result images is associated with an item listing for the object similar to the object of interest on an online marketplace.

16. The method of claim 5 , wherein the user-provided image is a user-captured image.

17. The method of claim 5 , wherein pre-processing the user-provided image comprises:
performing a person segmentation process on the user-provided image to produce a segmented image;
disposing one or more image segments of the segmented image on a background to produce an intermediate image;
performing a human pose estimation process on the intermediate image to identify a plurality of points of interest in the intermediate image, wherein the plurality of points of interest is associated with the object of interest; and
cropping the intermediate image based on the plurality of points of interest to produce the pre-processed image.

18. The non-transitory computer-readable medium of claim 8 , wherein the user-provided image is a user-captured image.

19. The non-transitory computer-readable medium of claim 8 , wherein a first search result image of the set of one or more search result images is a stock-quality image.

20. The non-transitory computer-readable medium of claim 8 , wherein pre-processing the user-provided image comprises:
performing a person segmentation process on the user-provided image to produce a segmented image;
disposing one or more image segments of the segmented image on a background to produce an intermediate image;
performing a human pose estimation process on the intermediate image to identify a plurality of points of interest in the intermediate image,

Assignees

• Ebay Inc

Inventors

• Kiapour Mohammadhadi
• Zheng Shuai
• Piramuthu Robinson
• Poursaeed Omid

Classifications

• G06Q30/0625 Primary

  by formulating product or service queries, e.g. using keywords or predefined options · CPC title

• G06F16/532

  Query formulation, e.g. graphical querying · CPC title

• G06F16/5838

  using colour · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title