Generation of stylized drawing of three-dimensional shapes using neural networks

Patent metadata

Field	Value
Publication number	US-11880913-B2
Application number	US-202117452568-A
Country	US
Kind code	B2
Filing date	Oct 27, 2021
Priority date	Oct 11, 2021
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

Techniques for generating a stylized drawing of three-dimensional (3D) shapes using neural networks are disclosed. A processing device generates a set of vector curve paths from a viewpoint of a 3D shape; extracts, using a first neural network of a plurality of neural networks of a machine learning model, surface geometry features of the 3D shape based on geometric properties of surface points of the 3D shape; determines, using a second neural network of the plurality of neural networks of the machine learning model, a set of at least one predicted stroke attribute based on the surface geometry features and a predetermined drawing style; generates, based on the at least one predicted stroke attribute, a set of vector stroke paths corresponding to the set of vector curve paths; and outputs a two-dimensional (2D) stylized stroke drawing of the 3D shape based at least on the set of vector stroke paths.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
a memory component storing instructions configured to, when executed by the one or more processors, cause the one or more processors to perform operations including:
generating a set of vector curve paths from a viewpoint of a three-dimensional (3D) shape;
extracting, using a first neural network of a plurality of neural networks of a machine learning model, surface geometry features of the 3D shape based on geometric properties of surface points of the 3D shape;
determining, using a second neural network of the plurality of neural networks of the machine learning model, a set of at least one predicted stroke attribute based at least on the surface geometry features and a predetermined drawing style;
generating, based on the at least one predicted stroke attribute, a set of vector stroke paths corresponding to the set of vector curve paths; and
outputting a two-dimensional (2D) stylized stroke drawing of the 3D shape based at least on the set of vector stroke paths.

2. The system of claim 1 , wherein:
each vector curve path of the set of vector curve paths is represented using a plurality of control points; and
the set of at least one predicted stroke attribute includes at least one predicted stroke attribute for each of the plurality of control points of each vector curve path.

3. The system of claim 2 , wherein the at least one predicted stroke attribute for each of the plurality of control points includes at least one of a corresponding thickness or a corresponding displacement.

4. The system of claim 3 , wherein the generating the set of vector stroke paths includes applying, to each of the plurality of control points of each vector curve path, the corresponding thickness and the corresponding displacement.

5. The system of claim 1 , wherein the surface geometry features from which the set of at least one predicted stroke attribute is generated includes one or more of a depth of the 3D shape associated with the viewpoint, radial curvature of the 3D shape, maximum principal surface curvature of the 3D shape, minimum principal surface curvature of the 3D shape, view-dependent surface curvature of the 3D shape, a dot product of surface normal with view vector associated with the viewpoint, or a binary image containing line segments of the vector curve paths.

6. The system of claim 1 , wherein the set of at least one predicted stroke attribute is determined further based on curve features of the vector curve paths including a tangent direction, a normal direction, and an arc length associated with each control point of the vector curve paths.

7. The system of claim 1 , wherein the machine learning model is trained with a training drawing and a representation of a corresponding training 3D shape.

8. The system of claim 1 , wherein the second neural network is implemented using a one-dimensional convolution neural network; and wherein the one-dimensional convolution neural network includes three layers, and outputs three channels including a first dimension of displacement, a second dimension of the displacement, and a thickness.

9. The system of claim 1 , wherein the operations further include rendering the set of vector stroke paths using a differential vector renderer to generate an untextured image;
wherein the 2D stylized stroke drawing is generated from the untextured image using a third neural network of the plurality of neural networks of the machine learning model; and
wherein the third neural network is a convolution neural network, and the untextured imaged is one of a plurality of input channels into the convolution neural network.

10. The system of claim 9 , wherein the plurality of input channels into the convolution neural network further includes one or more of a depth of the 3D shape associated with the viewpoint, radial curvature of the 3D shape, maximum principal surface curvature of the 3D shape, minimum principal surface curvature of the 3D shape, view-dependent surface curvature of the 3D shape, or a dot product of surface normal with view vector associated with the viewpoint.

11. A non-transitory computer-readable medium storing instructions configured to, when executed by one or more processors, cause the one or more processors to perform operations including:
generating a set of vector curve paths from a viewpoint of a three-dimensional (3D) shape;
extracting, using a first neural network of a plurality of neural networks of a machine learning model, surface geometry features of the 3D shape based on geometric properties of surface points of the 3D shape;
determining, using a second neural network of the plurality of neural networks of the machine learning model, a set of at least one predicted stroke attribute based at least on the surface geometry features and a predetermined drawing style;
generating, based on the at least one predicted stroke attribute, a set of vector stroke paths corresponding to the set of vector curve paths; and
outputting a two-dimensional (2D) stylized stroke drawing of the 3D shape based at least on the set of vector stroke paths.

12. The non-transitory computer-readable medium of claim 11 , wherein:
each vector curve path of the set of vector curve paths is represented using a plurality of control points; and
the set of at least one predicted stroke attribute includes at least one predicted stroke attribute for each of the plurality of control points of each vector curve path, the at least one predicted stroke attribute comprising at least one of a thickness or a displacement.

13. The non-transitory computer-readable medium of claim 11 , wherein the generating the set of vector stroke paths includes applying, to each of a plurality of control points of each of the set of vector curve paths, a corresponding thickness scalar and a corresponding displacement vector.

14. A computer-implemented method comprising:
generating a set of vector curve paths from a viewpoint of a three-dimensional (3D) shape;
extracting, using a first neural network of a plurality of neural networks of a machine learning model, surface geometry features of the 3D shape based on geometric properties of surface points of the 3D shape;
determining, using a second neural network of the plurality of neural networks of the machine learning model, a set of at least one predicted stroke attribute based at least on the surface geometry features and a predetermined drawing style;
generating, based on the at least one predicted stroke attribute, a set of vector stroke paths corresponding to the set of vector curve paths; and
outputting a two-dimensional (2D) stylized stroke drawing of the 3D shape based at least on the set of vector stroke paths.

15. The computer-implemented method of claim 14 , wherein:
each vector curve path of the set of vector curve paths is represented using a plurality of control points; and
the set of at least one predicted stroke attribute includes at least one predicted stroke attribute for each of the plurality of control points of each vector curve path, the at least one predicted stroke attribute comprising at least one of a thickness or a displacement.

16. The computer-implemented method of claim 14 , wherein the generating the set of vector stroke paths includes applying, to each of a plurality of control points of each of the set of vector curve paths, a corresponding thickness scalar and a corresponding displacement vector.

17. The computer-implemented method of claim 14 , wherein the set of at least one predicted stroke attribute is de

Assignees

• Adobe Inc
• Univ Massachusetts

Inventors

• Hertzmann Aaron
• Fisher Matthew
• Liu Difan
• Kalogerakis Evangelos

Classifications

• G06T11/23

  using straight lines or curves · CPC title

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T11/001 Primary

  Physics · mapped topic

• G06N3/045

  Combinations of networks · CPC title

• G06T11/203

  Physics · mapped topic