Method and apparatus for generating 3d hdr radiance fields

Patent metadata

Field	Value
Publication number	US-2024212115-A1
Application number	US-202318503784-A
Country	US
Kind code	A1
Filing date	Nov 7, 2023
Priority date	Dec 23, 2022
Publication date	Jun 27, 2024
Grant date	—

Abstract

An image processing device may perform a method for generating 3D HDR radiance fields, and the method may include: receiving an input of low dynamic range (LDR) images, which are captured at various viewpoints; constructing the 3D high dynamic range (HDR) radiance fields from the LDR images captured at the various viewpoints; generating an HDR image by ray marching on a basis of the 3D HDR radiance fields; generating a tone-mapped LDR image from the HDR image through a tone mapping module; calculating a loss value on a basis of the LDR images captured at the various viewpoints and the tone-mapped LDR image; and modifying parameters of the 3D HDR radiance fields and parameters of the tone-mapping module, so as to maximally minimize the loss value. The LDR images captured at the various viewpoints may be images captured under photometric conditions different from each other.

First claim

1 . A method for generating three-dimensional (3D) high dynamic range (HDR) radiance fields, the method comprising:
receiving, by an image processing device, an input of low dynamic range (LDR) images, which are captured at various viewpoints, as training data; constructing, by the image processing device, the 3D HDR radiance fields from the LDR images captured at the various viewpoints; generating, by the image processing device, an HDR image by ray marching on a basis of the 3D HDR radiance fields; generating, by the image processing device, a tone-mapped LDR image from the HDR image through a tone mapping module; calculating, by the image processing device, a loss value on a basis of the LDR images captured at the various viewpoints and the tone-mapped LDR image; and modifying, by the image processing device, parameters of the 3D HDR radiance fields and parameters of a tone-mapping module, so as to maximally minimize the loss value, wherein the LDR images captured at the various viewpoints are images captured under photometric conditions different from each other, and the tone-mapping module generates the tone-mapped LDR image by adjusting white balance of the HDR image and then applying a camera response function.

2 . The method of claim 1 , wherein the photometric conditions different from each other comprises at least one of an exposure value, the white balance, and the camera response function of an image capture device.

3 . The method of claim 1 , wherein the 3D HDR radiance fields are composed of voxels, and
a vertex located at each corner of the voxels stores spherical harmonic coefficients (SH coefficients) and opacity.

4 . The method of claim 3 , wherein a color value and opacity of a 3D point on a straight line is determined by performing trilinear interpolation for the spherical harmonic coefficients and opacity of the vertex located at each corner of the voxels.

5 . The method of claim 3 , wherein the modifying of the parameters comprises:
modifying the parameters after applying masking to the spherical harmonic coefficients, wherein in the masking, among the spherical harmonic coefficients, merely spherical harmonic coefficients of degrees less than or equal to a predetermined standard are modified in an early step of modifying the parameters of the 3D HDR radiance fields and spherical harmonic coefficients of degrees greater than or equal to a predetermined standard are also modified in a later step of modifying the 3D HDR radiance fields.

6 . The method of claim 1 , further comprising:
initializing, by the image processing device, the white balance of the generated HDR image, wherein the initializing of the white balance is to set a pixel value of the generated HDR image as a ratio value between an average color value of the entire HDR image and a color value of the generated HDR image.

7 . The method of claim 1 , wherein the modifying of the parameters comprises:
modifying parameters of a white balance function on a basis of a reference image, wherein among the input LDR images, the reference image is an image having a color value most similar to that of an average color of the input LDR images.

8 . The method of claim 1 , wherein the loss value comprises:
difference values between the tone-mapped LDR image and the LDR images captured at the various viewpoints; a value of voxels composed of the 3D HDR radiance fields not to differ much from those of neighboring voxels; and a value indicating how appropriate a tone-mapping camera response function is.

9 . An apparatus for generating three-dimensional (3D) high dynamic range (HDR) radiance fields, the apparatus comprising:
an input device configured to receive an input of low dynamic range (LDR) images, which are captured at various viewpoints, as training data; and a calculation device configured to generate the 3D HDR radiance fields from the LDR images captured at the various viewpoints, construct an HDR image by ray marching on a basis of the 3D HDR radiance fields, generate a tone-mapped LDR image from the HDR image through a tone mapping module, calculate a loss value on a basis of the LDR images captured at the various viewpoints and the tone-mapped LDR image, and modify parameters of the 3D HDR radiance fields and parameters of the tone mapping module, so as to maximally minimize the loss value, wherein the LDR images captured at the various viewpoints are images captured under photometric conditions different from each other, the 3D HDR radiance fields comprise information required for calculating how an object will appear on a screen when the object is viewed from a particular location, the ray marching determines a pixel value of the HDR image by using a color value and opacity of a 3D point on a straight line when the straight line is drawn in the 3D HDR radiance fields from a particular viewpoint, and the tone mapping module is configured to generate the tone-mapped LDR image by adjusting white balance of the HDR image and then applying a camera response function.

10 . The apparatus of claim 9 , wherein the photometric conditions different from each other comprise at least one of an exposure value, the white balance, and the camera response function of an image capture device.

11 . The apparatus of claim 9 , wherein the 3D HDR radiance fields are composed of voxels, and
a vertex located at each corner of the voxels is to store spherical harmonic coefficients (SH coefficients) and opacity.

12 . The apparatus of claim 11 , wherein the color value and opacity of the 3D point on the straight line are for being determined by performing trilinear interpolation for the spherical harmonic coefficients and opacity of the vertex located at each corner of the voxels.

13 . The apparatus of claim 11 , wherein the calculation device configured to modify the parameters comprises:
the calculation device configured to modify the parameters after applying masking to the spherical harmonic coefficients, wherein in the masking, among the spherical harmonic coefficients, merely spherical harmonic coefficients of degrees less than or equal to a predetermined standard are for being modified in an early step of modifying the parameters of the 3D HDR radiance fields and spherical harmonic coefficients of degrees greater than or equal to a predetermined standard are for being also modified in a later step of modifying the 3D HDR radiance fields.

14 . The apparatus of claim 9 , wherein the calculation device is configured to initialize the white balance of the constructed HDR image, and
the calculation device is configured to initialize the white balance to set the pixel value of the constructed HDR image as a ratio value between an average color value of the entire HDR image and a color value of the constructed HDR image.

15 . The apparatus of claim 9 , wherein the calculation device configured to modify the parameters comprises:
the calculation device configured to modify parameters of a white balance function on a basis of a reference image, wherein among the input LDR images, the reference image is an image having a color value most similar to that of an average color of the input LDR images.

16 . The apparatus of claim 9 , wherein the loss value comprises:
difference values between the tone-mapped LDR image and the LDR images captured at the various viewpoints; a value of voxels composed of the 3D HDR radiance fields-not to differ much from those of neighboring voxels; and a value indicating how appropriate a tone-mapping camera response function is.

17 .

Assignees

• Postech Res & Business Dev Found

Inventors

• Oh Tae Hyun
• Kim Jun Seong
• Kim Yu Ji
• Moon Ye Bin

Classifications

• G06T5/92 Primary

  based on global image properties · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title

• G06T15/08

  Volume rendering · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title