Hair rendering system based on deep neural network

Patent metadata

Field	Value
Publication number	US-12444128-B2
Application number	US-202118281966-A
Country	US
Kind code	B2
Filing date	Apr 13, 2021
Priority date	Apr 13, 2021
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

A deep neural network based hair rendering system is presented to model high frequency component of furry objects. Compared with existing approaches, the present method can generate photo-realistic rendering results. An acceleration method is applied in our framework, which can speed up training and rendering processes. In addition, a patch-based training scheme is introduced, which significantly increases the quality of outputs and preserves high frequency details.

First claim

What is claimed is:

1. A method of rendering an object, comprising:
capturing a plurality of images of the object from a plurality of views;
obtaining an alpha mask for each of the plurality of images;
obtaining a 3D proxy geometry through a structure from a silhouette algorithm;
rendering a depth map for each of the plurality of images using a 3D proxy;
training a neural radiance field-based (NeRF-based) deep neural network with images with depth; and
rendering a plurality of images for a plurality of new views.

2. The method according to claim 1 , further comprising:
training the NeRF-based deep neural network with pixels in an area of the image with depth.

3. The method according to claim 2 , wherein a near bound t n and a far bound t r of the area is calculated by:
t n =d s −d f *d r ; t f =d s +d f (1− d r ),

wherein d s is the depth of the image, d f is a sampling range, and d r is a ratio of sampling numbers.

4. The method according to claim 3 , wherein d f and d r are set based on a nearest depth and farthest depth rendered using the 3D proxy.

5. The method according to claim 2 , wherein the pixels in the images with depth used in training comprises no more than 10% of the pixels in the plurality of images.

6. The method according to claim 1 , further comprising:
dividing the images with depth into a plurality of patches; and
training NeRF-based deep neural network with the images with depth in the patch with a perceptual loss function for supervision.

7. The method of according to claim 1 , further comprising:
training the NeRF-based deep neural network using a red-green-blue (RGB) supervision and an alpha supervision using a loss function.

8. The method of according to claim 1 , wherein the NeRF-based deep neural network comprises a multi-layer UNet convolutional network.

9. The method according to claim 1 , wherein the NeRF-based deep neural network comprises a generative adversarial network, and the method further comprises:
training the generative adversarial network using a RGB supervision using a loss function.

10. The method according to claim 9 , further comprising:
training the generative adversarial network using the RGB supervision and a pseudo-supervision using the loss function.

11. The method according to claim 1 , wherein the object comprises hair.

12. A device for rendering an object, comprising:
a processor; and
a memory configured with computer instructions executable by the processor, wherein, upon being executed by the processor, the computer instructions cause the processor to perform operations, comprising:
obtaining an alpha mask for each of a plurality of images captured of the object from a plurality of views;
obtaining a 3D proxy geometry through a structure from a silhouette algorithm;

rendering a depth map for each of the plurality of images using a 3D proxy;
training a NeRF-based deep neural network with images with depth; and
rendering a plurality of images for a plurality of new views.

13. The device according to claim 12 , wherein the operation of training the NeRF-based deep neural network with the images with depth comprises:
training the NeRF-based deep neural network with pixels in an area of the image with depth.

14. The device according to claim 13 , wherein a near bound t n and a far bound t f of the area is calculated by:
t n =d s −d f *d r ; t f =d s +d f (1− d r ),

wherein d s is the depth of the image, d f is a sampling range, and d r is a ratio of sampling numbers.

15. The device according to claim 14 , wherein d f and d r are set based on a nearest depth and farthest depth rendered using the 3D proxy.

16. The device according to claim 13 , wherein the pixels in the images with depth used in training comprises no more than 10% of the pixels in the plurality of images.

17. The device according to claim 12 , wherein the operation of training the NeRF-based deep neural network with the images with depth comprises:
dividing the images with depth into a plurality of patches; and
training the NeRF-based deep neural network with the images with depth in the patch with a perceptual loss function for supervision.

18. The device according to claim 12 , wherein training the NeRF-based deep neural network with the images with depth comprises:
training the NeRF-based deep neural network using a RGB supervision and an alpha supervision using a loss function.

19. The device of according to claim 12 , wherein the NeRF-based deep neural network comprises a multi-layer UNet convolutional network.

20. The device of according to claim 12 , wherein the NeRF-based deep neural network comprises a generative adversarial network, and the operation of training the NeRF-based deep neural network with the images with depth comprises:
training the generative adversarial network using a RGB supervision using a loss function.

21. The device according to claim 20 , wherein the operation of training the NeRF-based deep neural network with the images with depth comprises:
training the generative adversarial network using the RGB supervision and a pseudo-supervision using the loss function.

22. The device according to claim 12 , wherein the object comprises hair.

Assignees

• Univ Shanghai Technology

Inventors

• Luo Haimin
• Wu Minye
• Xu Lan
• Yu Jingyi

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/596

  from three or more stereo images · CPC title