Methods, systems, and media for relighting images using predicted deep reflectance fields

Patent metadata

Field	Value
Publication number	US-10997457-B2
Application number	US-201916616235-A
Country	US
Kind code	B2
Filing date	Oct 16, 2019
Priority date	May 23, 2019
Publication date	May 4, 2021
Grant date	May 4, 2021

Abstract

Methods, systems, and media for relighting images using predicted deep reflectance fields are provided. In some embodiments, the method comprises: identifying a group of training samples, wherein each training sample includes (i) a group of one-light-at-a-time (OLAT) images that have each been captured when one light of a plurality of lights arranged on a lighting structure has been activated, (ii) a group of spherical color gradient images that have each been captured when the plurality of lights arranged on the lighting structure have been activated to each emit a particular color, and (iii) a lighting direction, wherein each image in the group of OLAT images and each of the spherical color gradient images are an image of a subject, and wherein the lighting direction indicates a relative orientation of a light to the subject; training a convolutional neural network using the group of training samples, wherein training the convolutional neural network comprises: for each training iteration in a series of training iterations and for each training sample in the group of training samples: generating an output predicted image, wherein the output predicted image is a representation of the subject associated with the training sample with lighting from the lighting direction associated with the training sample; identifying a ground-truth OLAT image included in the group of OLAT images for the training sample that corresponds to the lighting direction for the training sample; calculating a loss that indicates a perceptual difference between the output predicted image and the identified ground-truth OLAT image; and updating parameters of the convolutional neural network based on the calculated loss; identifying a test sample that includes a second group of spherical color gradient images and a second lighting direction; and generating a relit image of the subject included in each of the second group of spherical color gradient images with lighting from the second lighting direction using the trained convolutional neural network.

First claim

What is claimed is:

1. A method for relighting images using deep reflectance fields, comprising:
identifying a group of training samples, wherein each training sample includes (i) a group of one-light-at-a-time (OLAT) images that have each been captured when one light of a plurality of lights arranged on a lighting structure has been activated, (ii) a group of spherical color gradient images that have each been captured when the plurality of lights arranged on the lighting structure have been activated to each emit a particular color, and (iii) a lighting direction, wherein each image in the group of OLAT images and each of the spherical color gradient images are an image of a subject, and wherein the lighting direction indicates a relative orientation of a light to the subject;
training a convolutional neural network using the group of training samples, wherein training the convolutional neural network comprises:
for each training iteration in a series of training iterations and for each training sample in the group of training samples:
generating an output predicted image, wherein the output predicted image is a representation of the subject associated with the training sample with lighting from the lighting direction associated with the training sample;
identifying a ground-truth OLAT image included in the group of OLAT images for the training sample that corresponds to the lighting direction for the training sample;
calculating a loss that indicates a perceptual difference between the output predicted image and the identified ground-truth OLAT image; and
updating parameters of the convolutional neural network based on the calculated loss;

identifying a test sample that includes a second group of spherical color gradient images and a second lighting direction; and
generating a relit image of the subject included in each of the second group of spherical color gradient images with lighting from the second lighting direction using the trained convolutional neural network.

2. The method of claim 1 , wherein the lighting structure is a spherical dome, and wherein the plurality of lights are arranged on a surface of the spherical dome.

3. The method of claim 1 , wherein the loss is calculated using a pre-trained neural network.

4. The method of claim 1 , wherein the loss includes a first loss component that indicates the perceptual difference between the output image and the identified OLAT image based on texture information in each image, and wherein the loss includes a second loss component that indicates the perceptual difference between the output image and the identified OLAT image based on specularity information in each image.

5. The method of claim 4 , wherein the second loss component is calculated using a trained neural network that has been trained to take, as an input, an OLAT image, and to generate, as an output, a light direction of a light used to generate the OLAT image.

6. The method of claim 1 , wherein the group of OLAT images and the group of spherical color gradient images for each of the training samples are captured from a first plurality of cameras, each having a viewpoint from a first plurality of viewpoints, and wherein the second group of spherical color gradient images corresponding to the test sample are captured from a camera having a viewpoint that is not included in the first plurality of viewpoints.

7. The method of claim 1 , further comprising generating an aligned ground-truth OLAT image prior to calculating the loss, wherein the loss is calculated using the aligned ground-truth image.

8. A system for relighting images using deep reflectance fields, the system comprising:
a memory; and
a hardware processor that, when executing computer-executable instructions stored in the memory, is configured to:
identify a group of training samples, wherein each training sample includes (i) a group of one-light-at-a-time (OLAT) images that have each been captured when one light of a plurality of lights arranged on a lighting structure has been activated, (ii) a group of spherical color gradient images that have each been captured when the plurality of lights arranged on the lighting structure have been activated to each emit a particular color, and (iii) a lighting direction, wherein each image in the group of OLAT images and each of the spherical color gradient images are an image of a subject, and wherein the lighting direction indicates a relative orientation of a light to the subject;
train a convolutional neural network using the group of training samples, wherein training the convolutional neural network comprises:
for each training iteration in a series of training iterations and for each training sample in the group of training samples:
generating an output predicted image, wherein the output predicted image is a representation of the subject associated with the training sample with lighting from the lighting direction associated with the training sample;
identifying a ground-truth OLAT image included in the group of OLAT images for the training sample that corresponds to the lighting direction for the training sample;
calculating a loss that indicates a perceptual difference between the output predicted image and the identified ground-truth OLAT image; and
updating parameters of the convolutional neural network based on the calculated loss;

identify a test sample that includes a second group of spherical color gradient images and a second lighting direction; and
generate a relit image of the subject included in each of the second group of spherical color gradient images with lighting from the second lighting direction using the trained convolutional neural network.

9. The system of claim 8 , wherein the lighting structure is a spherical dome, and wherein the plurality of lights are arranged on a surface of the spherical dome.

10. The system of claim 8 , wherein the loss is calculated using a pre-trained neural network.

11. The system of claim 8 , wherein the loss includes a first loss component that indicates the perceptual difference between the output image and the identified OLAT image based on texture information in each image, and wherein the loss includes a second loss component that indicates the perceptual difference between the output image and the identified OLAT image based on specularity information in each image.

12. The system of claim 11 , wherein the second loss component is calculated using a trained neural network that has been trained to take, as an input, an OLAT image, and to generate, as an output, a light direction of a light used to generate the OLAT image.

13. The system of claim 8 , wherein the group of OLAT images and the group of spherical color gradient images for each of the training samples are captured from a first plurality of cameras, each having a viewpoint from a first plurality of viewpoints, and wherein the second group of spherical color gradient images corresponding to the test sample are captured from a camera having a viewpoint that is not included in the first plurality of viewpoints.

14. The system of claim 8 , wherein the hardware processor is further configured to generate an aligned ground-truth OLAT image prior to calculating the loss, wherein the loss is calculated using the aligned ground-truth image.

15. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for relighting images using deep reflectance fields, the method comprising:
identifying a group of training samples, wherein each training sample includes (i) a gro

Assignees

• Google Llc

Inventors

• Rhemann Christoph
• Meka Abhimitra
• Whalen Matthew
• Busch Jessica Lynn
• Bouaziz Sofien
• Harvey Geoffrey Douglas
• Tagliasacchi Andrea
• Taylor Jonathan
• Debevec Paul
• Denny Peter Joseph
• Fanello Sean Ryan Francesco
• Fyffe Graham
• Dourgarian Jason Angelo
• Yu Xueming
• Kowdle Adarsh Prakash Murthy
• Valentin Julien Pascal Christophe
• Lincoln Peter Christopher
• Pandey Rohit Kumar
• Häne Christian
• Izadi Shahram

Classifications

• G06T15/506 Primary

  Illumination models · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• G06N3/045

  Combinations of networks · CPC title