Utilizing an object relighting neural network to generate digital images illuminated from a target lighting direction

Patent metadata

Field	Value
Publication number	US-10692276-B2
Application number	US-201815970367-A
Country	US
Kind code	B2
Filing date	May 3, 2018
Priority date	May 3, 2018
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

The present disclosure relates to using an object relighting neural network to generate digital images portraying objects under target lighting directions based on sets of digital images portraying the objects under other lighting directions. For example, in one or more embodiments, the disclosed systems provide a sparse set of input digital images and a target lighting direction to an object relighting neural network. The disclosed systems then utilize the object relighting neural network to generate a target digital image that portrays the object illuminated by the target lighting direction. Using a plurality of target digital images, each portraying a different target lighting direction, the disclosed systems can also generate a modified digital image portraying the object illuminated by a target lighting configuration that comprises a combination of the different target lighting directions.

First claim

What is claimed is:

1. A non-transitory computer readable storage medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:
identify a set of input digital images portraying an object, the set of input digital images comprising a first input digital image portraying the object illuminated from a first lighting direction and a second input digital image portraying the object illuminated from a second lighting direction;
determine a target lighting direction different from the first lighting direction and the second lighting direction;
provide the set of input digital images to an object relighting neural network trained based on training digital images portraying objects illuminated by training lighting directions and ground truth digital images portraying the objects illuminated by additional lighting directions; and
utilize the object relighting neural network to generate a target digital image of the object illuminated from the target lighting direction based on the set of input digital images, the first lighting direction, and the second lighting direction.

2. The non-transitory computer readable storage medium of claim 1 , wherein the first lighting direction corresponds to a first light source and the second lighting direction corresponds to a second light source.

3. The non-transitory computer readable storage medium of claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
identify a first predetermined light direction range and a second predetermined light direction range;
select the first lighting direction by sampling from the first predetermined light direction range; and
select the second lighting direction by sampling from the second predetermined light direction range.

4. The non-transitory computer readable storage medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to provide the set of input digital images to the object relighting neural network by:
generating a first set of color channels comprising color values reflecting pixels of the first input digital image; and
generating a first set of direction channels comprising coordinates corresponding to the first lighting direction.

5. The non-transitory computer readable storage medium of claim 4 , further comprising instructions that, when executed by the at least one processor, cause the computing device to provide the set of input digital images to the object relighting neural network by:
generating a second set of color channels comprising color values reflecting pixels of the second input digital image;
generating a second set of direction channels comprising coordinates corresponding to the second lighting direction; and
providing the first set of color channels, the second set of color channels, the first set of direction channels, and the second set of direction channels to the object relighting neural network.

6. The non-transitory computer readable storage medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate a set of target direction channels comprising coordinates corresponding to the target lighting direction;
provide the set of target direction channels to the object relighting neural network; and
utilize the object relighting neural network to generate the target digital image further based on the set of target direction channels.

7. The non-transitory computer readable storage medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine an additional target lighting direction different from the first lighting direction, the second lighting direction, and the target lighting direction;
utilize the object relighting neural network to generate an additional target digital image of the object illuminated from the additional target lighting direction based on the set of input digital images, the first lighting direction, and the second lighting direction; and
based on the target digital image and the additional target digital image, generate a modified digital image portraying the object illuminated by a target light configuration comprising a first light source from the target lighting direction and a second light source from the additional target lighting direction.

8. The non-transitory computer readable storage medium of claim 1 , wherein the set of input digital images consists of five or fewer input digital images.

9. The non-transitory computer readable storage medium of claim 1 , wherein the object relighting neural network comprises a fully convolutional neural network.

10. The non-transitory computer readable storage medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to identify the set of input digital images portraying the object by:
illuminating the object from the first lighting direction;
capturing a first digital image of the object illuminated from the first lighting direction;
illuminating the object from the second lighting direction without the first lighting direction; and
capturing a second digital image of the object illuminated from the second lighting direction.

11. A method for generating digital images portraying objects under target lighting conditions based on prior digital images portraying objects under prior lighting conditions, comprising:
identifying a set of input digital images portraying an object, the set of input digital images comprising a first input digital image portraying the object illuminated from a first lighting direction and a second input digital image portraying the object illuminated from a second lighting direction;
determining a target lighting direction different from the first lighting direction and the second lighting direction;
providing the set of input digital images to an object relighting neural network trained based on training digital images portraying objects illuminated by training lighting directions and ground truth digital images portraying the objects illuminated by additional lighting directions; and
utilizing the object relighting neural network to generate a target digital image of the object illuminated from the target lighting direction based on the set of input digital images, the first lighting direction, and the second lighting direction.

12. The method of claim 11 , wherein the first lighting direction corresponds to a first light source and the second lighting direction corresponds to a second light source.

13. The method of claim 12 , further comprising:
identifying a first predetermined light direction range and a second predetermined light direction range;
selecting the first lighting direction by sampling from the first predetermined light direction range; and
selecting the second lighting direction by sampling from the second predetermined light direction range.

14. The method of claim 11 , wherein providing the set of input digital images to an object relighting neural network comprises:
generating a first set of color channels comprising color values reflecting pixels of the first input digital image; and
generating a first set of direction channels comprising coordinates corresponding to the first lighting direction.

15. The method of claim 14 , wherein providing the set of input digital images to an object relighting neural network comprises:
generating a second set of

Assignees

• Adobe Inc

Inventors

• Sunkavalli Kalyan
• Xu Zexiang
• Hadap Sunil

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title