Neural compositing by embedding generative technologies into non-destructive document editing workflows

Patent metadata

Field	Value
Publication number	US-2024135611-A1
Application number	US-202318188671-A
Country	US
Kind code	A1
Filing date	Mar 23, 2023
Priority date	Oct 17, 2022
Publication date	Apr 25, 2024
Grant date	—

Abstract

One or more aspects of the method, apparatus, and non-transitory computer readable medium include obtaining an original image, a scene graph describing elements of the original image, and a description of a modification to the original image. The one or more aspects further include updating the scene graph based on the description of the modification. The one or more aspects further include generating a modified image using an image generation neural network based on the updated scene graph, wherein the modified image incorporates content based on the original image and the description of the modification.

First claim

What is claimed is:

1 . A method for neural compositing, comprising:
obtaining an original image, a scene graph describing elements of the original image, and a description of a modification to the original image; updating the scene graph based on the description of the modification; and generating a modified image using an image generation neural network based on the updated scene graph, wherein the modified image incorporates content based on the original image and the description of the modification.

2 . The method of claim 1 , further comprising:
updating the scene graph by adding a node to the scene graph based on the description of the modification.

3 . The method of claim 1 , further comprising:
updating the scene graph by modifying a node of the scene graph based on the description of the modification.

4 . The method of claim 1 , further comprising:
generating a partial image corresponding to a node of the updated scene graph; and combining the partial image with the original image to obtain the modified image.

5 . The method of claim 4 , further comprising:
generating a seed for the partial image, wherein the partial image is based on the seed, and wherein the node includes the seed as an attribute.

6 . The method of claim 5 , further comprising:
generating an additional partial image corresponding to the node based on the seed; and combining the additional partial image with the modified image to obtain an additional modified image.

7 . The method of claim 4 , further comprising:
identifying one or more conditions for generating the partial image, wherein the node includes the one or more conditions as attributes.

8 . The method of claim 7 , further comprising:
generating an additional partial image corresponding to the node based on the one or more conditions; and combining the additional partial image with the modified image to obtain an additional modified image.

9 . The method of claim 1 , wherein:
the image generation neural network is trained to generate the modified image using a diffusion training technique.

10 . The method of claim 1 , further comprising:
receiving an additional description of an additional modification to the original image; updating the scene graph based on the description of the additional modification; and generating an additional modified image using the image generation neural network based on the updated scene graph, wherein the additional modified image incorporates content based on the original image and the additional description of the additional modification.

11 . The method of claim 10 , further comprising:
generating a first partial image based on a description of the modification, wherein the modified image is based on the first partial image; generating a second partial image based on a second description of a second modification, wherein the additional modified image is based on the second partial image; and resampling the first partial image based on the description and the second partial image.

12 . An apparatus for neural compositing, comprising:
a processor; and a memory storing instructions and in electronic communication with the processor, the processor being configured to execute the instructions to: obtain an original image, a scene graph describing elements of the original image, and a description of a modification to the original image; update the scene graph based on the description of the modification; and generate a modified image using an image generation neural network based on the updated scene graph, wherein the modified image incorporates content based on the original image and the description of the modification.

13 . The apparatus of claim 12 , the processor being further configured to execute the instructions to:
update the scene graph by adding a node to the scene graph based on the description of the modification.

14 . The apparatus of claim 12 , the processor being further configured to execute the instructions to:
update the scene graph by modifying a node of the scene graph based on the description of the modification.

15 . The apparatus of claim 12 , the processor being further configured to execute the instructions to:
generate a partial image corresponding to a node of the updated scene graph; and combine the partial image with the original image to obtain the modified image.

16 . The apparatus of claim 15 , the processor being further configured to execute the instructions to:
generate a seed for the partial image, wherein the partial image is based on the seed, and wherein the node includes the seed as an attribute.

17 . A non-transitory computer readable medium storing code for image generation, the code comprising instructions executable by a processor to:
obtaining an original image, a scene graph describing elements of the original image, and a description of a modification to the original image; update the scene graph based on the description of the modification; and generate a modified image using an image generation neural network based on the updated scene graph, wherein the modified image incorporates content based on the original image and the description of the modification.

18 . The non-transitory computer readable medium of claim 17 , the code further comprising instructions executable by the processor to:
update the scene graph by adding a node to the scene graph based on the description of the modification.

19 . The non-transitory computer readable medium of claim 17 , the code further comprising instructions executable by the processor to:
update the scene graph by modifying a node of the scene graph based on the description of the modification.

20 . The non-transitory computer readable medium of claim 17 , the code further comprising instructions executable by the processor to:
generate a partial image corresponding to a node of the updated scene graph; and combine the partial image with the original image to obtain the modified image.

Assignees

• Adobe Inc

Inventors

• Costin Alexandru Vasile
• Brdiczka Oliver
• Darabi Aliakbar
• Brown Davis Taylor
• Bourgin David Davenport

Classifications

• G06V10/82

  using neural networks · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T2207/20081

  Training; Learning · CPC title