Asynchronous multi-user real-time streaming of web-based image edits using generative adversarial network(s)

Patent metadata

Field	Value
Publication number	US-11928762-B2
Application number	US-202117466699-A
Country	US
Kind code	B2
Filing date	Sep 3, 2021
Priority date	Sep 3, 2021
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

Embodiments of the present invention provide systems, methods, and computer storage media for editing images using a web-based intermediary between a user interface on a client device and an image editing neural network(s) (e.g., a generative adversarial network) on a server(s). The present image editing system supports multiple users in the same software container, advanced concurrency of projection and transformation of the same image, clubbing transformation requests from several users hosted in the same software container, and smooth display updates during a progressive projection.

First claim

What is claimed is:

1. One or more computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
triggering a multi-iteration projection of an image into a latent space using a first neural network to extract a latent vector representing the image, the multi-iteration projection comprising non-final iterations that progressively update a non-optimized interim version of the latent vector prior to optimization resulting in a final optimized version of the latent vector;
storing the non-optimized interim version of the latent vector in a persistent layer of a database;
storing the non-optimized interim version from the persistent layer in a non-persistent layer of the database; and
in response to a user edit via a user interface, triggering a second neural network to generate a transformed version of the image using the non-optimized interim version of the latent vector representing the image from the non-persistent layer simultaneously as the first neural network projects the image into the latent space to extract the latent vector representing the image.

2. The one or more computer storage media of claim 1 , the operations further comprising periodically storing the non-optimized interim version of the latent vector in the persistent layer after every completion of a number of the non-final iterations of the multi-iteration projection.

3. The one or more computer storage media of claim 1 , wherein the operations are associated with a session created for a web user on a particular worker in a multi-process container on a server, the particular worker supporting multiple instances of the operations for multiple web users.

4. The one or more computer storage media of claim 1 , the operations further comprising clubbing transformation requests from multiple users hosted in a same software container prior to triggering the second neural network to generate the transformed version of the image.

5. The one or more computer storage media of claim 1 , the operations further comprising clubbing transformation requests from multiple users hosted in a same software container into an array of target latent vectors, wherein triggering the second neural network comprises transmitting the array to the second neural network.

6. The one or more computer storage media of claim 1 , the operations further comprising aggregating target latent vectors generated during multiple sessions running for multiple users on a common worker in a common software container.

7. The one or more computer storage media of claim 1 , the operations further comprising periodically causing display of an approximation of the image corresponding to the non-optimized interim version of the latent vector, using an arithmetic sequence to increment a step size between iterations of the multi-iteration projection that trigger successive updates to the display.

8. A computerized method comprising:
triggering an optimization of a latent vector representing an image, the optimization comprising a plurality of iterations that use a first neural network to progressively update a non-optimized interim version of the latent vector;
storing the non-optimized interim version of the latent vector in a persistent layer of a database;
storing the non-optimized interim version of the latent vector from the persistent layer in a non-persistent layer of the database; and
generating an edit to the image, during the optimization of the latent vector, by triggering a second neural network to generate a transformed version of the image using the non-optimized interim version of the latent vector from the non-persistent layer simultaneously as the first neural network progressively updates the non-optimized interim version of the latent vector representing the image.

9. The computerized method of claim 8 , further comprising periodically storing the non-optimized interim version of the latent vector in the persistent layer after every completion of a number of iterations of the optimization.

10. The computerized method of claim 8 , wherein the computerized method is associated with a session created for a web user on a particular worker in a multi-process container on a server, the particular worker supporting multiple instances of method for multiple web users.

11. The computerized method of claim 8 , further comprising clubbing transformation requests from multiple users hosted in a same software container prior to triggering the second neural network to generate the transformed version of the image.

12. The computerized method of claim 8 , further comprising clubbing transformation requests from multiple users hosted in a same software container into an array of target latent vectors, wherein triggering the second neural network comprises transmitting the array to the second neural network.

13. The computerized method of claim 8 , further comprising aggregating target latent vectors generated during multiple sessions running for multiple users on a common worker in a common software container.

14. The computerized method of claim 8 , further comprising periodically causing display of an approximation of the image corresponding to the non-optimized interim version of the latent vector, using an arithmetic sequence to increment a step size between iterations of the optimization that trigger successive updates to the display.

15. A computer system comprising:
memory comprising a persistent layer and a non-persistent layer; and
one or more hardware processors configured to cause the computer system to perform operations comprising:
triggering a projection of an image using a first neural network to progressively update a non-optimized interim version of a latent vector representing the image;
storing the non-optimized interim version of the latent vector in the persistent layer;
storing the non-optimized interim version from the persistent layer in the non-persistent layer; and
in response to a user edit via a user interface, triggering, during the projection of the image, a second neural network to generate a transformed version of the image using the non-optimized interim version of the latent vector from the non-persistent layer simultaneously as the first neural network progressively updates the non-optimized interim version of the latent vector representing the image.

16. The computer system of claim 15 , the operations further comprising periodically storing the non-optimized interim version of the latent vector in the persistent layer after every completion of a number of iterations of projection.

17. The computer system of claim 15 , wherein the operations are associated with a session created for a web user on a particular worker in a multi-process software container executing on the one or more hardware processors, the particular worker supporting multiple instances of the operations for multiple web users.

18. The computer system of claim 15 , the operations further comprising clubbing transformation requests from multiple users hosted in a same software container prior to triggering the second neural network to generate the transformed version of the image.

19. The computer system of claim 15 , the operations further comprising clubbing transformation requests from multiple users hosted in a same software container into an array of target latent vectors, wherein triggering the second neural network comprises transmitting the array to the second neural network.

Assignees

• Adobe Inc

Inventors

• Kumar Akhilesh
• Kalarot Ratheesh
• Faieta Baldo
• Ghadar Shabnam

Classifications

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T3/4046

  using neural networks · CPC title

• H04L65/4015

  where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title