Indoor scene virtual roaming method based on reflection decomposition

Patent metadata

Field	Value
Publication number	US-12406438-B2
Application number	US-202318490790-A
Country	US
Kind code	B2
Filing date	Oct 20, 2023
Priority date	Apr 21, 2021
Publication date	Sep 2, 2025
Grant date	Sep 2, 2025

Abstract

Disclosed in the present invention is an indoor scene virtual roaming method based on reflection decomposition, the method includes: firstly, by means of three-dimensional reconstruction, obtaining a rough global triangular mesh model projection as an initial depth map, aligning depth edges to color edges, and converting the aligned depth map into a simplified triangular mesh; checking planes in the global triangular mesh model, and if a certain plane is a reflection plane, constructing a double-layer expression in a reflection area for each picture in which the reflection plane is visible, so as to correctly render the reflection effect on an object surface; and giving a virtual viewport, using neighborhood pictures and the triangular mesh to draw a picture of the virtual viewport, and for the reflection area, using foreground and background pictures and foreground and background triangular meshes to perform drawing.

First claim

What is claimed is:

1. An indoor scene virtual roaming method based on reflection decomposition, comprising:
step S 1 , capturing pictures sufficient for covering a target indoor scene, carrying out three-dimensional reconstruction for the target indoor scene based on the captured pictures, and obtaining internal and external camera parameters and a global triangular mesh model of the target indoor scene;
step S 2 , for each picture, projecting the global triangular mesh model into a corresponding depth map, aligning depth edges to color edges, converting the aligned depth map into a triangular mesh, and performing mesh simplification on the triangular mesh;
step S 3 , detecting a plane in the global triangular mesh model, and detecting whether the plane is a reflection plane by means of color consistency between adjacent images; when the plane is the reflection plane, constructing a double-layer expression on a reflection area for each picture in which the reflection plane is visible to correctly render a reflection effect of an object surface; wherein the double-layer expression comprises double-layer triangular meshes of foreground and background and two decomposed pictures of foreground and background, wherein a foreground triangular mesh is used for expressing object surface geometry, and a background triangular mesh is used for expressing a mirror image of a scene geometry on the reflection plane; a foreground picture is used for expressing object surface textures after removing reflection components, and a background picture is used for expressing the reflection components of the scene on the object surface; step S 3 comprises the following sub-steps:
sub-step S 31 , detecting planes in the global triangular mesh model, reserving planes with an area larger than an area threshold, projecting the planes onto visible pictures, and recording a set of pictures in which the planes are visible as ; for each picture I k in , calculating a set k of K neighboring pictures thereof, wherein a calculation of K neighbors is obtained according to an ordering of overlapping rate of vertices in the global triangular mesh model after plane reflection;
constructing a matching cost volume using k , determining whether the plane has enough reflection components in the picture I k , wherein a determining method is as follows: for each pixel, after mirroring the global triangular mesh model according to a plane equation, finding a cost corresponding to a mirrored depth value in the matching cost volume, and determining whether a cost position is a local minimum point; when a number of pixels of the local minimum points of a cost in the picture is greater than a pixel number threshold, determining that the plane has reflection components in the picture; when a number of visible pictures with reflection components in a certain plane is greater than a picture number threshold, determining the plane to be the reflection plane;
sub-step S 32 , for each reflection plane, calculating a two-dimensional reflection area β k thereof on each visible picture, sub-step S 32 comprises: projecting the reflection plane onto the visible picture to obtain a projected depth map, expanding the projected depth map, and comparing the expanded projected depth map with the aligned depth map to obtain an accurate two-dimensional reflection area; screening each pixel with a depth value in the projected depth map by three-dimensional point distances and normal included angles, and taking a screened pixel area as the two-dimensional reflection area β k of the reflection plane on the picture;
sub-step S 33 , constructing the double-layer expression on the reflection area for each picture in which the reflection plane is visible, sub-step S 33 comprises: taking the projected depth map as an initial foreground depth map, mirroring internal and external parameters of the camera of the picture into a virtual camera according to the plane equation, rendering an initial background depth map in the virtual camera by using the global triangular mesh model, and converting the initial foreground depth map and the initial background depth map into simplified two layers of triangular meshes M k 0 and M k 1 ;
calculating two layers of foreground and background pictures I k 0 and I k 1 by an iterative optimization algorithm, and further optimizing M k 0 and M k 1 , wherein all related original pictures are subjected to an inverse gamma correction in advance for subsequent decomposition;
an optimization objective is to minimize the following energy function:

arg
⁢
min

(

R
,
T

)

k
1

,

M
k

0
,
1

,

I
k

0
,
1

⁢

E
d

+

λ
s

⁢

E
s

+

λ
p

⁢

E
p

,

s
.
t
.

I
k

0
,
1

(
u
)

∈

[

0
,
1

]

where (R, T) k 1 in the optimization objective represents a rigid body transformation of a triangular mesh of a reflection layer, and initial values thereof are identity matrix and 0, respectively, and M k 0 and M k 1 only optimize three-dimensional positions of mesh vertexes without changing topological structures; E d , E s and E p are a data item, a smoothing term and a prior term, respectively, λ s , λ p are weights of respective items, and u represents a pixel in I k 0,1 ; the following relations are satisfied:

E
d

=

Assignees

• Univ Zhejiang

Inventors

• Xu Weiwei
• Xu Jiamin
• Wu Xiuchao
• Zhu Zihan
• Bao Hujun

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T7/40

  Analysis of texture (depth or shape recovery from texture G06T7/529) · CPC title

• G06T5/92

  based on global image properties · CPC title

• G06V10/56

  relating to colour · CPC title