Dynamic lighting capture and reconstruction

Patent metadata

Field	Value
Publication number	US-9854176-B2
Application number	US-201414164006-A
Country	US
Kind code	B2
Filing date	Jan 24, 2014
Priority date	Jan 24, 2014
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

Systems and techniques for dynamically capturing and reconstructing lighting are provided. The systems and techniques may be based on a stream of images capturing the lighting within an environment as a scene is shot. Reconstructed lighting data may be used to illuminate a character in a computer-generated environment as the scene is shot. For example, a method may include receiving a stream of images representing lighting of a physical environment. The method may further include compressing the stream of images to reduce an amount of data used in reconstructing the lighting of the physical environment and may further include outputting the compressed stream of images for reconstructing the lighting of the physical environment using the compressed stream, the reconstructed lighting being used to render a computer-generated environment.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving a stream of one or more images of a physical environment;
generating one or more light maps based on the one or more images, wherein the one or more light maps include a plurality of light intensity values indicating lighting of the physical environment;
determining a plurality of coefficients representing the plurality of light intensity values of the one or more light maps;
generating one or more compressed light maps, the one or more compressed light maps including the plurality of coefficients representing the plurality of light intensity values of the one or more light maps;
comparing the plurality of coefficients with stored coefficients generated for one or more previous light maps to identify one or more coefficients of the plurality of coefficients that have changed as compared with the stored coefficients of the one or more previous light maps, wherein a number of coefficients in the identified one or more coefficients is less than a number of coefficients in the plurality of coefficients; and
transmitting, to a rendering device, only the identified one or more coefficients of the plurality of coefficients for reconstructing the lighting of the physical environment, the reconstructed lighting being usable by the rendering device to render a computer-generated environment with the lighting of the physical environment.

2. The method of claim 1 , wherein compressing the stream of the one or more images comprises:
detecting a change in an image of the one or more images compared to one or more previously received images;
determining whether the detected change is below a threshold level; and
declining to use the image for reconstructing the lighting of the physical environment in response to determining that the detected change is below the threshold level.

3. The method of claim 1 , wherein the lighting of the physical environment is reconstructed using the identified one or more coefficients.

4. The method of claim 3 , wherein the reconstructed lighting of the physical environment includes a light map representing the reconstructed lighting of the physical environment.

5. The method of claim 1 , wherein the reconstructed lighting of the physical environment includes one or more reconstruction coefficients representing the reconstructed lighting of the physical environment.

6. The method of claim 1 , wherein the one or more coefficients include one or more spherical harmonic coefficients.

7. The method of claim 1 , wherein a coefficient of the plurality of one or more coefficients includes one or more wavelet basis coefficient coefficients.

8. The method of claim 1 , wherein the one or more images include low dynamic range images.

9. The method of claim 8 , wherein the one or more images include a first low dynamic range image captured at a first exposure value, a second low dynamic range image captured at a second exposure value different than the first exposure value, and a third dynamic range image captured at a third exposure value different than the first and second exposure values.

10. The method of claim 9 , wherein generating the one or more light maps includes combining the first low dynamic range image, the second low dynamic range image, and the third dynamic range image.

11. The method of claim 1 , wherein the one or more light maps include a high dynamic range image representing the lighting of the physical environment with a greater dynamic range of luminosity than the one or more images.

12. A computer-implemented method, comprising:
receiving a stream of one or more images of a physical environment;
generating one or more light maps based on the one or more images;
compressing the one or more light maps to produce one or more compressed light maps, the one or more compressed light maps including a plurality of coefficients associated with the one or more light maps;
comparing the plurality of coefficients with stored coefficients generated for one or more previous light maps to identify one or more coefficients of the plurality of coefficients that have changed as compared with the stored coefficients of the one or more previous light maps, wherein a number of coefficients in the identified one or more coefficients is less than a number of coefficients in the plurality of coefficients; and
transmitting only the identified one or more coefficients of the plurality of coefficients for reconstructing the lighting of the physical environment.

13. A system, comprising:
a memory storing a plurality of instructions; and
one or more processors to:
receive a stream of one or more images of a physical environment;
generate one or more light maps based on the one or more images, wherein the one or more light maps include a plurality of light intensity values indicating lighting of the physical environment;
determine a plurality of coefficients representing the plurality of light intensity values of the one or more light maps;
generate one or more compressed light maps, the one or more compressed light maps including the plurality of coefficients representing the plurality of light intensity values of the one or more light maps;
compare the plurality of coefficients with stored coefficients generated for one or more previous light maps to identify one or more coefficients of the plurality of coefficients that have changed as compared with the stored coefficients of the one or more previous light maps, wherein a number of coefficients in the identified one or more coefficients is less than a number of coefficients in the plurality of coefficients; and
transmit, to a rendering device, only the identified one or more coefficients of the plurality of coefficient for reconstructing the lighting of the physical environment, the reconstructed lighting being usable by the rendering device to render a computer-generated environment with the lighting of the physical environment.

14. The system of claim 13 , wherein compressing the stream of the one or more images comprises:
detecting a change in an image of the one or more images compared to one or more previously received images;
determining whether the detected change is below a threshold level; and
declining to use the image for reconstructing the lighting of the physical environment in response to determining that the detected change is below the threshold level.

15. A computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:
instructions that cause the one or more processors to receive a stream of one or more images of a physical environment;
instructions that cause the one or more processors to generate one or more light maps based on the one or more images, wherein the one or more light maps include a plurality of light intensity values indicating lighting of the physical environment;
instructions that cause the one or more processors to determine a plurality of coefficients representing the plurality of light intensity values of the one or more light maps;
instructions that cause the one or more processors to generate one or more compressed light maps including the plurality of coefficients representing the plurality of light intensity values of the one or more light maps;
instructions that cause the one or more processors to compare the plurality of coefficients with stored coefficients generated for one or more previous light maps to identify one or more coefficients of the plurality of coefficients that have changed as compared with the stored coefficients of the one o

Assignees

• Lucasfilm Entertainment Co Ltd

Inventors

• Sanders Michael
• Bhat Kiran
• Miyashiro Curt Isamu
• Snell Jason
• Grabli Stephane

Classifications

• H04N23/73

  by influencing the exposure time · CPC title

• H04N23/71 Primary

  Circuitry for evaluating the brightness variation · CPC title

• H04N19/63

  using sub-band based transform, e.g. wavelets · CPC title

• H04N19/61

  in combination with predictive coding · CPC title

• G06T15/50

  Lighting effects · CPC title