Three-dimensional reconstruction and angle of view synthesis method for moving human body

Patent metadata

Field	Value
Publication number	US-12518485-B2
Application number	US-202318331972-A
Country	US
Kind code	B2
Filing date	Jun 9, 2023
Priority date	Dec 10, 2020
Publication date	Jan 6, 2026
Grant date	Jan 6, 2026

Abstract

Disclosed in the present invention is a three-dimensional reconstruction and angle of view synthesis method for a moving human body, which performs reconstruction of a moving human body by optimizing three-dimensional representations of the moving human body from an inputted multi-angle of view video. The method provided by the present invention comprises: defining a set of hidden variables on mesh vertices of a deformable human body model; transforming, on the basis of the deformation characteristics of the human body model, the set of structured hidden variables to a position of a target human body posture; mapping, on the basis of a neural network, the set of structured hidden variables to continuous voxel density and color for representing the geometric and appearance of the human body; and optimizing, on the basis of differentiable rendering, a neural network implicit function in an inputted multi-angle of view video; and performing three-dimensional reconstruction and angle of view synthesis of the moving human body on the basis of the optimized neural network implicit function. By means of a neural network implicit function, the present invention implements three-dimensional reconstruction and angle of view synthesis of a moving human body at a very small number of angles of view, which is the first method for achieving high-quality angle of view synthesis at a very small number of angles of view.

First claim

What is claimed is:

1 . A method for three-dimensional reconstruction and view synthesis of a dynamic human body, comprising steps of:
(1) attaching latent variables to mesh nodes of a human model to construct a set of structured latent variables; (2) constructing a neural network implicit function on the basis of the structured latent variables to represent geometry and appearance of the human body; (3) rendering, by a differentiable volume renderer, the neural network implicit function into a two-dimensional image, and optimizing a representation of the neural network implicit function by minimizing an error between a corresponding frame and a corresponding view image in the rendered image and a multi-view video; and (4) performing three-dimensional reconstruction and view synthesis of the dynamic human body based on the optimized neural network implicit function.

2 . The method for three-dimensional reconstruction and view synthesis of a dynamic human body according to claim 1 , wherein in step (1), the human body model is a deformable human body model, and the mesh nodes of the deformable human body model are driven by a posture of the human body to change a spatial position of the constructed structured latent variables.

3 . The method for three-dimensional reconstruction and view synthesis of a dynamic human body according to claim 1 , wherein in step (2), the step of constructing a neural network implicit function on the basis of the structured latent variables to represent geometry and appearance of the human body comprises: taking the structured latent variables as a local latent variable, assigning a latent variable to any point in a three-dimensional space by a latent variable diffusion method, and regressing to a volume density and a color by the neural network implicit function.

4 . The method for three-dimensional reconstruction and view synthesis of a dynamic human body according to claim 3 , wherein the latent variable diffusion method comprises: directly performing interpolation or taking nearest neighbor values for the structured latent variables, or processing the structured latent variables by using a three-dimensional network to allow interaction of information among the latent variables, and then performing trilinear interpolation or taking nearest neighbor values for the latent variables processed by the network to obtain corresponding latent variables, wherein the three-dimensional network is a point cloud processing network or a three-dimensional convolution network.

5 . The method for three-dimensional reconstruction and view synthesis of a dynamic human body according to claim 1 , wherein in step (3), the step of rendering, by a differentiable volume renderer, the neural network implicit function into a two-dimensional image comprises: sampling a set of three-dimensional points along light projected to a pixel by a camera, calculating a volume density and a color of the three-dimensional points by using the neural network implicit function, and accumulating the volume density and the color on the light to obtain a pixel color.

6 . The method for three-dimensional reconstruction and view synthesis of a dynamic human body according to claim 1 , wherein in step (4), the three-dimensional reconstruction of the human body is realized by extracting a human mesh model from the optimized neural network implicit function by a Marching cubes algorithm, and the view synthesis is realized by obtaining a two-dimensional image by using the differentiable volume renderer.

Assignees

• Univ Zhejiang

Inventors

• Zhou Xiaowei
• Bao Hujun
• Peng sida

Classifications

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06V10/82

  using neural networks · CPC title

• H04N13/351

  for displaying simultaneously · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title