Apparatus and method for synthesizing 2D image and media using inverse rendering with 3D spatially varying lighting information estimation

Patent metadata

Field	Value
Publication number	US-12548237-B2
Application number	US-202318393078-A
Country	US
Kind code	B2
Filing date	Dec 21, 2023
Priority date	Nov 15, 2023
Publication date	Feb 10, 2026
Grant date	Feb 10, 2026

Abstract

The present disclosure relates to an apparatus for synthesizing a 2D image and media using a reverse rendering including 3D spatially varying lighting information estimation, and the apparatus includes a target view analysis unit configured to estimate a normal map and direct lighting using a plurality of 2D images, a material estimation unit configured to reflect the normal map and direct lighting estimated from the target view analysis unit and estimate material information, a 3D lighting estimation unit configured to reflect the direct lighting estimated from the target view analysis unit and the material information estimated from the material estimation unit and estimate 3D spatially varying lighting information, and an image generation unit configured to reflect the estimated shapes, materials, and 3D spatially varying lighting information, synthesize the 2D images with 3D objects, and generate a synthesized image.

First claim

The invention claimed is:

1 . An apparatus for synthesizing a 2D image and media using a reverse rendering including 3D spatially varying lighting information estimation, the apparatus comprising:
a target view analysis unit configured to estimate a normal map and direct lighting using a plurality of 2D images; a material estimation unit configured to reflect the normal map and direct lighting estimated from the target view analysis unit and estimate material information; a 3D lighting estimation unit configured to reflect the direct lighting estimated from the target view analysis unit and the material information estimated from the material estimation unit and estimate 3D spatially varying lighting information; and an image generation unit configured to reflect the estimated shapes, materials, and 3D spatially varying lighting information, synthesize the 2D images with 3D objects, and generate a synthesized image.

2 . The apparatus of claim 1 , wherein the target view analysis unit includes:
a normal map estimation unit configured to receive RGB images acquired from the plurality of 2D images, a depth map, and a confidence map and estimate a normal map from the depth map; an incident lighting estimation unit configured to estimate incident direct lighting using the RGB images, the depth map, the confidence map, and the normal map; and an exitant lighting estimation unit configured to estimate exitant direct lighting using the RGB images, the depth map, the confidence map, and the normal map.

3 . The apparatus of claim 1 , wherein the material estimation unit includes:
a specular radiance estimation unit configured to estimate a specular radiance feature (f spec ) based on the normal map and an incident direct lighting estimated value (η, λ, ξ) estimated from the target view analysis unit; a context estimation unit configured to estimate a context feature (f context ) based on the RGB images, the depth map, and the confidence map; a multi-view aggregation unit configured to aggregate the specular radiance feature (f spec ), the context feature (f context ), and multi-view RGB images and generate a multi-view aggregation feature; and an albedo and roughness estimation unit configured to estimate albedo and roughness based on the RGB images, the depth map, the confidence map, the context feature (f context ), and a bidirectional reflectance distribution function (BRDF) feature (f BRDF ) output from the multi-view aggregation unit.

4 . The apparatus of claim 1 , wherein the 3D lighting estimation unit reflects the RGB images, the depth map, the confidence map, an exitant direct lighting volume estimated from the target view analysis unit, and the albedo and roughness information estimated from the material estimation unit, estimates the 3D spatially varying lighting information, and outputs the 3D spatially varying lighting volume.

5 . The apparatus of claim 1 , wherein the image generation unit reflects the shapes, materials, and 3D spatially varying lighting information estimated from the 3D lighting estimation unit and generates a novel view image.

6 . The apparatus of claim 1 , wherein the image generation unit reflects the shapes, materials, and 3D spatially varying lighting information estimated from the 3D lighting estimation unit, synthesizes sounds or 3D objects and the sounds into the 2D images, and generates a synthesized image.

7 . A method of synthesizing a 2D image and media using a reverse rendering including 3D spatially varying lighting information estimation, the method comprising:
a target view analyzing operation of estimating a normal map and direct lighting using a plurality of 2D images; a material estimating operation of reflecting the estimated normal map and direct lighting and estimating material information; a 3D lighting estimating operation of reflecting the estimated direct lighting and material information and estimating 3D spatially varying lighting information; and an image generating operation of reflecting the estimated shapes, materials, and 3D spatially varying lighting information, synthesizing the 2D images with 3D objects, and generating an image.

8 . The method of claim 7 , wherein the target view analyzing operation includes:
receiving RGB images acquired from the plurality of 2D images, a depth map, and a confidence map; estimating a normal map from the depth map; estimating incident direct lighting using the RGB images, the depth map, the confidence map, and the normal map; and estimating exitant direct lighting using the RGB images, the depth map, the confidence map, and the normal map.

9 . The method of claim 7 , wherein the material estimating operation includes:
estimating a specular radiance feature (f spec ) based on the normal map and an incident direct lighting estimated value (η, λ, ξ) estimated from the target view analyzing operation; estimating a context feature (f context ) based on the RGB images, the depth map, and the confidence map; aggregating the specular radiance feature (f spec ), the context feature (f context ), and multi-view RGB images and generating a multi-view aggregation feature; and estimating albedo and roughness based on the RGB images, the depth map, the confidence map, the context feature (f context ), and a bidirectional reflectance distribution function (BRDF) feature (f BRDF ) output from a multi-view aggregation unit configured to generate the multi-view aggregation feature.

10 . The method of claim 7 , wherein the 3D lighting estimating operation includes reflecting the RGB images, the depth map, the confidence map, an exitant direct lighting volume estimated from the target view analyzing operation, and the albedo and roughness information estimated from the material estimating operation, estimating the 3D spatially varying lighting information, and outputting the 3D spatially varying lighting volume.

11 . The method of claim 7 , wherein the image generating operation includes:
setting positions of the 3D objects to be synthesized into a 2D image when the target 2D image and the 3D objects are input; and calculating lighting information at the set positions of the 3D objects using the 3D spatially varying lighting information volume acquired through the 3D lighting estimating operation and generating an image by rendering the 3D objects including shadows based on the calculated lighting information.

12 . The method of claim 11 , wherein the image generating operation further includes generating a novel view image using the 3D spatially varying lighting volume acquired through the 3D lighting estimating operation.

13 . The method of claim 7 , wherein the image generating operation includes reflecting the shapes, materials, and 3D spatially varying lighting information estimated from the 3D lighting estimating operation, synthesizing sounds or 3D objects and the sounds into the 2D images, and generating a synthesized image.

Assignees

• Korea Inst Sci & Tech

Inventors

• Cho Junghyun
• Kim Ig Jae
• Park Haesol
• Choi Jun Yong
• Lee Seok Yeong

Classifications

• G06T15/506 Primary

  Illumination models · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T15/87

  Gouraud shading · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title