Relightable texture for use in rendering an image

Patent metadata

Field	Value
Publication number	US-9607429-B2
Application number	US-201414121221-A
Country	US
Kind code	B2
Filing date	Aug 13, 2014
Priority date	Jun 4, 2014
Publication date	Mar 28, 2017
Grant date	Mar 28, 2017

Abstract

Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. An initial texture can be segmented into materials and an initial coarse color estimate is determined for each material. Scene geometry is estimated from the captured views of the scene and is used to scale the initial coarse color estimates relative to each other such that the different materials appear to be lit with a similar irradiance. In this way, a global irradiance function is estimated describing the scene illumination. This provides a starting point for a color estimate and shading estimate extraction. The shading estimate can be used to fit surface normals to the global irradiance function. The set of surface normals and the color estimate are stored for subsequent use to allow relighting of the scene.

First claim

The invention claimed is:

1. A method of determining a colour component of a relightable texture and a set of surface normals for use in rendering an image from a rendering viewpoint under arbitrary lighting conditions, wherein at least one view of a scene from a respective at least one camera viewpoint represents the image, the method comprising:
analysing the at least one view of the scene to estimate scene geometry and to segment an initial texture into a plurality of materials, the initial texture being separable into a colour estimate and a corresponding shading estimate;
determining an initial coarse colour estimate for each of the materials;
determining one or more scale factors, for scaling a respective one or more of the initial coarse colour estimates, the scale factors being determined based on differences between irradiance estimates determined for the materials based on shading estimates which correspond with scaled versions of the initial coarse colour estimates of the materials;
determining a global irradiance function for the scene using the determined scale factors;
using the global irradiance function and the initial texture to determine a further colour estimate and a corresponding further shading estimate, wherein the further colour estimate represents the colour component of the relightable texture; and
determining the set of surface normals using the global irradiance function and the further shading estimate.

2. The method of claim 1 wherein each of the initial coarse colour estimates comprises a single value for the respective material.

3. The method of claim 1 wherein, for each of the materials, the initial coarse colour estimate is determined by finding the average of the values in the initial texture for the material.

4. The method of claim 1 wherein the further colour estimate and the further shading estimate have per-texel values.

5. The method of claim 1 wherein the set of surface normals comprises a surface normal for each of a plurality of sample positions on a surface of the scene geometry.

6. The method of any claim 1 wherein said determining a global irradiance function for the scene using the determined scale factors comprises:
scaling the one or more of the initial coarse colour estimates using the determined scale factors; and
using the scaled coarse colour estimates to determine the global irradiance function.

7. The method of claim 6 wherein said using the scaled coarse colour estimates to determine the global irradiance function comprises:
combining the scaled coarse colour estimates to form a global coarse colour estimate; and
using the global coarse colour estimate to determine the global irradiance function.

8. The method of claim 7 wherein said using the global coarse colour estimate to determine the global irradiance function comprises:
dividing the initial texture by the global coarse colour estimate to determine a global irradiance estimate; and
determining the global irradiance function by finding the best fit to the global irradiance estimate of spherical harmonics up to the second order.

9. The method of claim 1 wherein said determining one or more scale factors comprises:
determining a measure of the differences between irradiance estimates for different materials in overlapping regions of the irradiance estimates, said measure of the differences being dependent upon the scale factors, wherein for a particular scale factor the irradiance estimate for a material matches the shading estimate which corresponds with a version of the initial coarse colour estimate scaled with the particular scale factor; and
determining the scale factors so as to minimise the measure of the differences.

10. The method of claim 1 wherein said using the global irradiance function and the initial texture to determine a further colour estimate comprises:
dividing the initial texture by the global irradiance function to determine an intermediate colour estimate; and
filtering the intermediate colour estimate with a bilateral filter to determine said further colour estimate.

11. The method of claim 10 wherein the bilateral filter uses a measure of similarity between texels based on both: (i) the similarity in luminance of the texels, and (ii) the similarity in chrominance of the texels.

12. The method of claim 11 wherein the further colour estimate, A(x), has values at texel positions, x, and wherein the bilateral filter is described by the equation:

A
⁡

(
x
)

=

1
u

⁢

∫
μ

⁢

W
⁡

(
μ
)

⁢

ⅇ

-



x
-
μ



2
2

σ
w
2

⁢

ⅇ

-

(

cos

-
1

⁡

(

W
⋒

⁡

(
x
)

T

⁢

W
⋒

⁡

(

Assignees

• Imagination Tech Ltd

Inventors

• Imber James
• Hilton Adrian

Classifications

• G06T7/49

  based on structural texture description, e.g. using primitives or placement rules · CPC title

• G06T15/506 Primary

  Illumination models · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T15/50

  Lighting effects · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title