Editing digital images utilizing a neural network with an in-network rendering layer

Patent metadata

Field	Value
Publication number	US-10430978-B2
Application number	US-201715448206-A
Country	US
Kind code	B2
Filing date	Mar 2, 2017
Priority date	Mar 2, 2017
Publication date	Oct 1, 2019
Grant date	Oct 1, 2019

Abstract

The present disclosure includes methods and systems for generating modified digital images utilizing a neural network that includes a rendering layer. In particular, the disclosed systems and methods can train a neural network to decompose an input digital image into intrinsic physical properties (e.g., such as material, illumination, and shape). Moreover, the systems and methods can substitute one of the intrinsic physical properties for a target property (e.g., a modified material, illumination, or shape). The systems and methods can utilize a rendering layer trained to synthesize a digital image to generate a modified digital image based on the target property and the remaining (unsubstituted) intrinsic physical properties. Systems and methods can increase the accuracy of modified digital images by generating modified digital images that realistically reflect a confluence of intrinsic physical properties of an input digital image and target (i.e., modified) properties.

First claim

We claim:

1. A system for generating modified digital images from input digital images, comprising:
one or more memories, comprising:
a neural network comprising a rendering layer trained to generate synthesized digital images from input digital images portraying diffuse materials and input digital images portraying specular materials; and
an input digital image; and

at least one computing device storing instructions thereon, that, when executed by the at least one computing device, cause the system to:
predict, utilizing the neural network, a material property set, a surface orientation map, and an illumination environment map based on the input digital image;
replace at least one of the material property set, the surface orientation map, or the illumination environment map with a target material property set, a target surface orientation map, or a target illumination map; and
utilize the rendering layer of the neural network to generate a modified digital image from the input digital image based on at least one of the target material property set, the target surface orientation map, or the target illumination map and at least two of the material property set, the surface orientation map, or the illumination environment map.

2. The system of claim 1 , further comprising instructions that, when executed by the at least one computing device, cause the system to:
replace the material property set with the target material property set; and
utilize the rendering layer of the neural network to generate the modified digital image based on the target material property, the surface orientation map, and the illumination environment map.

3. The system of claim 1 , wherein the input digital image portrays a first material corresponding to the material property set and portrays a second material, and further comprising instructions that, when executed by the at least one computing device, further cause the system to:
predict, utilizing the neural network, the material property set corresponding to the first material and a second material property set corresponding to the second material;
replace the material property set with the target material property set;
replace the second material property set with a second target material property set; and
utilize the rendering layer of the neural network to generate the modified digital image based on the target material property set, the second target material property set, the target surface orientation map, and the target illumination map.

4. The system of claim 1 , further comprising instructions that, when executed by the at least one computing device, cause the system to predict, utilizing the neural network, the material property set based on a parametric representation that is differentiable with respect to incoming light directions, outgoing light directions, and material properties.

5. The system of claim 1 , further comprising instructions that, when executed by the at least one computing device, cause the system to predict, utilizing the neural network, the surface orientation map, wherein the surface orientation map comprises an RGB digital image having a plurality of pixels with RBG values, where the RBG value of each pixel of the plurality of pixel encodes x, y, and z dimensions of a surface orientation of each pixel of an object portrayed in the digital image.

6. The system of claim 1 , further comprising instructions that, when executed by the at least one computing device, cause the system to generate the modified digital image such that the modified digital image portrays a specular material.

7. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:
receive an input digital image portraying an object within an illumination environment, wherein the object comprises a surface of a material, and wherein the surface has a surface normal direction;
predict, utilizing a neural network comprising a rendering layer trained to generate synthesized digital images from input digital images portraying diffuse materials and input digital images portraying specular materials, a material property set, a surface orientation map, and an illumination environment map based on the input digital image;
replace at least one of the material property set, the surface orientation map, or the illumination environment map with a target material property set, a target surface orientation map, or a target illumination map; and
utilize the rendering layer of the neural network to generate a modified digital image from the input digital image based on at least one of the target material property set, the target surface orientation map, or the target illumination map and at least two of the material property set, the surface orientation map, or the illumination environment map.

8. The non-transitory computer readable medium of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
replace the at least one of the material property set, the surface orientation map, or the illumination environment map by replacing the material property set with the target material property set; and
utilize the rendering layer of the neural network to generate the modified digital image by generating the modified digital image based on the target material property set, the surface orientation map, and the illumination environment map.

9. The non-transitory computer readable medium of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
replace the at least one of the material property set, the surface orientation map, or the illumination environment map by replacing the illumination environment map with the target illumination map; and
utilize the rendering layer of the neural network to generate the modified digital image by generating the modified digital image based on the material property set, the surface orientation map, and the target illumination map.

10. The non-transitory computer readable medium of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
receive a request to modify the surface normal direction by replacing the object with a second object that comprises a second surface of a second material having a second normal direction; and
wherein the modified digital image portrays the second normal direction of the second object with the material of the object in the illumination environment of the input digital image.

11. The non-transitory computer readable medium of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
replace the at least one of the material property set, the surface orientation map, or the illumination environment map by replacing the surface orientation map with the target surface orientation map; and
utilize the rendering layer of the neural network to generate the modified digital image by generating the modified digital image based on the material property set, the target surface orientation map, and the illumination environment map.

12. The non-transitory computer readable medium claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computer system to generate the modified digital image such that the modified digital image portrays a specular material.

13. The non-transitory computer readable medium of claim 7 , further comprising instructions that, when executed by the at least one pr

Assignees

• Adobe Inc

Inventors

• Yumer Mehmet
• Yang Jimei
• Liu Guilin
• Ceylan Aksit Duygu

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F30/00

  Computer-aided design [CAD] · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06T15/205

  Image-based rendering · CPC title