Web-based digital image editing in real time utilizing a latent vector stream renderer and an image modification neural network

Patent metadata

Field	Value
Publication number	US-11727614-B2
Application number	US-202117182492-A
Country	US
Kind code	B2
Filing date	Feb 23, 2021
Priority date	Feb 23, 2021
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

The present disclosure describes systems, methods, and non-transitory computer readable media for detecting user interactions to edit a digital image from a client device and modify the digital image for the client device by using a web-based intermediary that modifies a latent vector of the digital image and an image modification neural network to generate a modified digital image from the modified latent vector. In response to user interaction to modify a digital image, for instance, the disclosed systems modify a latent vector extracted from the digital image to reflect the requested modification. The disclosed systems further use a latent vector stream renderer (as an intermediary device) to generate an image delta that indicates a difference between the digital image and the modified digital image. The disclosed systems then provide the image delta as part of a digital stream to a client device to quickly render the modified digital image.

First claim

What is claimed is:

1. A non-transitory computer readable medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
extracting a latent image vector from an initial digital image displayed via a client device;
receiving an indication of a user interaction to modify the initial digital image;
in response to the indication of the user interaction, determining an image-differential metric reflecting a difference between the initial digital image and a modified digital image generated by an image modification neural network based on a change within the latent image vector corresponding to the user interaction; and
providing the image-differential metric to the client device for rendering the modified digital image.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the processing device, cause the processing device to perform operations comprising:
based on the latent image vector, providing an initial digital stream to the client device to cause the client device to display the initial digital image; and
providing the image-differential metric as part of a modified digital stream to the client device to cause the client device to display the modified digital image in place of the initial digital image.

3. The non-transitory computer readable medium of claim 1 , wherein determining the image-differential metric comprises:
modifying the latent image vector utilizing a first computing device in response to the indication of the user interaction to modify the initial digital image;
generating the modified digital image from the modified latent image vector utilizing the image modification neural network on a second computing device; and
determining the difference between the initial digital image and the modified digital image utilizing the first computing device.

4. The non-transitory computer readable medium of claim 3 , wherein modifying the latent image vector comprises:
determining a vector direction and a measure of change corresponding to the vector direction associated with the user interaction; and
modifying the latent image vector in the vector direction according to the measure of change.

5. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the processing device, cause the processing device to perform operations comprising:
modifying the latent image vector in a vector direction corresponding to the user interaction; and
generating the modified digital image from the modified latent image vector by performing a GAN-based operation utilizing the image modification neural network.

6. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the processing device, cause the processing device to perform operations comprising:
receiving the indication of the user interaction to modify the initial digital image by receiving an indication of a user interaction selecting one or more additional digital images to combine with the initial digital image;
generating a modified latent image vector by combining the latent image vector with one or more additional latent image vectors corresponding to the one or more additional digital images; and
based on the modified latent image vector, generating the modified digital image depicting a combination of image features from the initial digital image and the one or more additional digital images utilizing the image modification neural network.

7. The non-transitory computer readable medium of claim 6 , further comprising instructions that, when executed by the processing device, cause the processing device to perform operations comprising:
receiving the indication of the user interaction to modify the initial digital image by receiving an indication of a user interaction to modify an image feature of the initial digital image;
modifying the latent image vector to represent the modified image feature in a modified latent feature vector; and
generating the modified digital image based on the modified latent feature vector utilizing a generative adversarial neural network (GAN) as the image modification neural network.

8. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the processing device, cause the processing device to perform operations comprising:
providing, for display on the client device, an image modification interface comprising a sketch tool for drawing strokes on the initial digital image; and
based on receiving an indication of strokes drawn on the initial digital image, providing the image-differential metric to the client device for rendering the modified digital image comprising an overlay of additional image features corresponding to the strokes.

9. A system comprising:
one or more memory devices comprising an image modification neural network; and
one or more processors that are configured to cause the system to:
extract an initial latent image vector from an initial digital image displayed via a client device;
receive an indication of a user interaction to modify the initial digital image;
in response to the indication of the user interaction, generate an image-differential metric reflecting a difference between the initial digital image and a modified digital image by:
generating, utilizing the image modification neural network, the modified digital image reflecting an image modification to the initial digital image based on a change within the initial latent image vector corresponding to the user interaction; and
determining the difference between the initial digital image and the modified digital image; and

provide the image-differential metric to the client device for rendering the modified digital image.

10. The system of claim 9 , wherein the one or more processors are further configured to cause the system to:
receive an additional indication of an additional user interaction to modify the initial digital image;
in response to the additional indication of the additional user interaction, generate an additional image-differential metric indicating a difference between the modified digital image and a further modified digital image; and
provide the additional image-differential metric as part of a digital stream to the client device for rendering the further modified digital image in place of the modified digital image.

11. The system of claim 9 , wherein the one or more processors are further configured to cause the system to:
extract the initial latent image vector from the initial digital image displayed via a mobile device as the client device; and
provide the image-differential metric to the mobile device in response to the user interaction.

12. The system of claim 9 , wherein the one or more processors are further configured to cause the system to:
generate the modified digital image by processing a modified latent image vector corresponding to the change within the initial latent image vector utilizing the image modification neural network on a server device at a first location;
generate the image-differential metric by determining the difference between the initial digital image and the modified digital image via a computing device at a second location; and
provide the image-differential metric for rendering the modified digital image within a browser on the client device at a third location.

13. The system of claim 9 , wherein the one or more processors are further configured to cause the system to:
provide, for display

Assignees

• Adobe Inc

Inventors

• Kumar Akhilesh
• Faieta Baldo
• Walczyszyn Piotr
• Kalarot Ratheesh
• Bagnall Archie
• Ghadar Shabnam
• Lin Wei-An
• Smith Cameron
• Cantrell Christian
• Hebron Patrick
• Chan Wilson
• Lu Jingwan
• Winnemoeller Holger
• Olsen Sven

Classifications

• G06T11/23

  using straight lines or curves · CPC title

• G06T11/26 Primary

  Drawing of charts or graphs · CPC title

• G06T3/04

  Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12) · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title