Single image three-dimensional hair reconstruction

Patent metadata

Field	Value
Publication number	US-12374036-B2
Application number	US-202217814063-A
Country	US
Kind code	B2
Filing date	Jul 21, 2022
Priority date	Jul 21, 2022
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

A system to enable 3D hair reconstruction and rendering from a single reference image which performs a multi-stage process that utilizes both a 3D implicit representation and a 2D parametric embedding space.

First claim

What is claimed is:

1. A method comprising:
accessing an input image, the input image comprising a set of image features that depict a display of an object;
detecting an obstructed portion of the object based on the display of the object;
generating an estimation of the obstructed portion based on a neural network;
generating a three-dimensional (3D) shape based on the set of image features that depict the object and the estimation of the obstructed portion;
generating a U, V coordinate system (UV) texture map based on the input image and the 3D shape;
generating a 3D model based on the 3D shape and the UV texture map; and
causing display of a presentation of the 3D model at a position within a target image.

2. The method of claim 1 , wherein the generating the 3D shape based on the set of image features that depict the object further comprises:
extracting a set of global features and a set of local features from the input image;
performing a pixel-aligned implicit function based on the set of global features and the set of local features; and
generating the 3D shape based on the pixel-aligned implicit function.

3. The method of claim 1 , wherein the generating the UV texture map based on the input image and the 3D shape further comprises:
generating a projection based on the input image;
generating a segmentation mask based on a portion of the 3D shape; and
generating the UV texture map based on the projection and the segmentation mask.

4. The method of claim 1 , wherein the causing display of the presentation of the 3D model at the position within the target image further comprises:
determining a set of canonical coordinates of the 3D model based on the input image; and
causing display of the presentation of the 3D model at the position within the target image based on the canonical coordinates.

5. The method of claim 1 , wherein the object depicted in the input image is a first object, and the causing display of the presentation of the 3D model at the position within the target image further comprises:
identifying a second object within the target image; and
causing display of the presentation of the 3D model at the position within the target image based on the second object.

6. The method of claim 5 , wherein the causing display of the presentation of the 3D model at the position within the target image further comprises:
adjusting a scale of the 3D model based on a size of the second object within the target image.

7. The method of claim 1 , wherein the object includes a human head.

8. A system comprising:
one or more processors; and
a memory comprising instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
accessing an input image, the input image comprising a set of image features that depict a display of an object;
detecting an obstructed portion of the object based on the display of the object;
generating an estimation of the obstructed portion based on a neural network;
generating a three-dimensional (3D) shape based on the set of image features that depict the object and the estimation of the obstructed portion;
generating a U, V coordinate system (UV) texture map based on the input image and the 3D shape;
generating a 3D model based on the 3D shape and the UV texture map; and
causing display of a presentation of the 3D model at a position within a target image.

9. The system of claim 8 , wherein the generating the 3D shape based on the set of image features that depict the object further comprises:
extracting a set of global features and a set of local features from the input image;
performing a pixel-aligned implicit function based on the set of global features and the set of local features; and
generating the 3D shape based on the pixel-aligned implicit function.

10. The system of claim 8 , wherein the generating the UV texture map based on the input image and the 3D shape further comprises:
generating a projection based on the input image;
generating a segmentation mask based on a portion of the 3D shape; and
generating the UV texture map based on the projection and the segmentation mask.

11. The system of claim 8 , wherein the causing display of the presentation of the 3D model at the position within the target image further comprises:
determining a set of canonical coordinates of the 3D model based on the input image; and
causing display of the presentation of the 3D model at the position within the target image based on the canonical coordinates.

12. The system of claim 8 , wherein the object depicted in the input image is a first object, and the causing display of the presentation of the 3D model at the position within the target image further comprises:
identifying a second object within the target image; and
causing display of the presentation of the 3D model at the position within the target image based on the second object.

13. The system of claim 5 , wherein the causing display of the presentation of the 3D model at the position within the target image further comprises:
adjusting a scale of the 3D model based on a size of the second object within the target image.

14. The wherein the object of claim 8 , wherein the object includes a human head.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
accessing an input image, the input image comprising a set of image features that depict a display of an object;
detecting an obstructed portion of the object based on the display of the object;
generating an estimation of the obstructed portion based on a neural network;
generating a three-dimensional (3D) shape based on the set of image features that depict the object and the estimation of the obstructed portion;
generating a U, V coordinate system (UV) texture map based on the input image and the 3D shape;
generating a 3D model based on the 3D shape and the UV texture map; and
causing display of a presentation of the 3D model at a position within a target image.

16. The non-transitory machine-readable storage medium of claim 15 , wherein the generating the 3D shape based on the set of image features that depict the object further comprises:
extracting a set of global features and a set of local features from the input image;
performing a pixel-aligned implicit function based on the set of global features and the set of local features; and
generating the 3D shape based on the pixel-aligned implicit function.

17. The non-transitory machine-readable storage medium of claim 15 , wherein the generating the UV texture map based on the input image and the 3D shape further comprises:
generating a projection based on the input image;
generating a segmentation mask based on a portion of the 3D shape; and
generating the UV texture map based on the projection and the segmentation mask.

18. The non-transitory machine-readable storage medium of claim 15 , wherein the causing display of the presentation of the 3D model at the position within the target image further comprises:
determining a set of canonical coordinates of the 3D model based on the input image; and
causing display of the presentation of the 3D model at the position within the target image based on the canonical coordinates.

19. The non-transitory machine-readable storage medium of claim 15 , wherein the object depicted in the input image is a first object, and the ca

Assignees

• Snap Inc

Inventors

• Huang Zeng
• Chai Menglei
• Tulyakov Sergey
• Olszewski Kyle
• Lee Hsin-Ying

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title