Generation of stylized drawing of three- dimensional shapes using neural networks

Patent metadata

Field	Value
Publication number	US-12536715-B2
Application number	US-202418419287-A
Country	US
Kind code	B2
Filing date	Jan 22, 2024
Priority date	Oct 11, 2021
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Techniques for generating a stylized drawing of three-dimensional (3D) shapes using neural networks are disclosed. A processing device generates a set of vector curve paths from a viewpoint of a 3D shape; extracts, using a first neural network of a plurality of neural networks of a machine learning model, surface geometry features of the 3D shape based on geometric properties of surface points of the 3D shape; determines, using a second neural network of the plurality of neural networks of the machine learning model, a set of at least one predicted stroke attribute based on the surface geometry features and a predetermined drawing style; generates, based on the at least one predicted stroke attribute, a set of vector stroke paths corresponding to the set of vector curve paths; and outputs a two-dimensional (2D) stylized stroke drawing of the 3D shape based at least on the set of vector stroke paths.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
receiving a training drawing and a representation of a corresponding training three-dimensional (3D) shape into a machine learning model; and training the machine learning model based on the training drawing and the corresponding training 3D shape to generate a trained machine learning model that generates a two-dimensional (2D) stylized stroke drawing of an input 3D shape based at least on a style of the training drawing and surface geometry features of the input 3D shape.

2 . The computer-implemented method of claim 1 , wherein training the machine learning model includes:
generating a binary mask of the training drawing; using the machine learning model to generate an untextured image based on the training 3D shape; using the machine learning model to generate a textured image based on the untextured image; and applying a loss function comprising a plurality of loss terms including:
a first loss term that compares portions of the untextured image with portions of the binary mask; and
a second loss term that compares portions of the textured image with portions of the training drawing.

3 . The computer-implemented method of claim 2 , wherein the plurality of loss terms further includes a third loss term comprising a shape regularization term on predicted displacements used to generate the untextured image.

4 . The computer-implemented method of claim 3 , wherein the plurality of loss terms further includes a fourth loss term comprising an adversarial loss associated with a discriminator.

5 . The computer-implemented method of claim 4 , wherein the machine learning model includes three neural networks that are trained collectively using a weighted sum of the first loss term, the second loss term, the third loss term, and the fourth loss term.

6 . The computer-implemented method of claim 2 , wherein the untextured image includes a rasterized image, and the textured image includes a predicted stylized image in RGB (red, green, blue) space.

7 . The computer-implemented method of claim 2 , further comprising:
determining a plurality of crop sizes; and generating the portions of the training drawing, the portions of the binary mask, the portions of the untextured image, and the portions of the textured image according to the plurality of crop sizes.

8 . The computer-implemented method of claim 1 , wherein the machine learning model is trained using a single training drawing.

9 . The computer-implemented method of claim 1 , wherein the generated trained machine learning model is configured to:
generate a set of vector curve paths from a viewpoint of a three-dimensional (3D) shape; extract, using a first neural network of a plurality of neural networks associated with the trained machine learning model, surface geometry features of the 3D shape based on geometric properties of surface points of the 3D shape; determine, using a second neural network of the plurality of neural networks, a set of at least one predicted stroke attribute based at least on the surface geometry features and a predetermined drawing style; generate, based on the at least one predicted stroke attribute, a set of vector stroke paths corresponding to the set of vector curve paths; and output a two-dimensional (2D) stylized stroke drawing of the 3D shape based at least on the set of vector stroke paths.

10 . A non-transitory computer-readable medium storing instructions configured to, when executed by one or more processors, cause the one or more processors to perform operations including:
receiving a training drawing and a representation of a corresponding training three-dimensional (3D) shape into a machine learning model; and training the machine learning model based on the training drawing and the corresponding training 3D shape to generate a trained machine learning model that generates a two-dimensional (2D) stylized stroke drawing of an input 3D shape based at least on a style of the training drawing and surface geometry features of the input 3D shape.

11 . The non-transitory computer-readable medium of claim 10 , wherein training the machine learning model includes:
generating a binary mask of the training drawing; using the machine learning model to generate an untextured image based on the training 3D shape; using the machine learning model to generate a textured image based on the untextured image; and applying a loss function comprising a plurality of loss terms including:
a first loss term that compares portions of the untextured image with portions of the binary mask; and
a second loss term that compares portions of the textured image with portions of the training drawing.

12 . The non-transitory computer-readable medium of claim 11 , wherein the plurality of loss terms further includes a third loss term comprising a shape regularization term on predicted displacements used to generate the untextured image.

13 . The non-transitory computer-readable medium of claim 12 , wherein the plurality of loss terms further includes a fourth loss term comprising an adversarial loss associated with a discriminator.

14 . The non-transitory computer-readable medium of claim 13 , wherein the machine learning model includes three neural networks that are trained collectively using a weighted sum of the first loss term, the second loss term, the third loss term, and the fourth loss term.

15 . The non-transitory computer-readable medium of claim 11 , further comprising:
determining a plurality of crop sizes; and generating the portions of the training drawing, the portions of the binary mask, the portions of the untextured image, and the portions of the textured image according to the plurality of crop sizes.

16 . The non-transitory computer-readable medium of claim 10 , wherein the machine learning model is trained using a single training drawing.

17 . The non-transitory computer-readable medium of claim 10 , wherein the generated trained machine learning model is configured to:
generate a set of vector curve paths from a viewpoint of a three-dimensional (3D) shape; extract, using a first neural network of a plurality of neural networks associated with the trained machine learning model, surface geometry features of the 3D shape based on geometric properties of surface points of the 3D shape; determine, using a second neural network of the plurality of neural networks, a set of at least one predicted stroke attribute based at least on the surface geometry features and a predetermined drawing style; generate, based on the at least one predicted stroke attribute, a set of vector stroke paths corresponding to the set of vector curve paths; and output a two-dimensional (2D) stylized stroke drawing of the 3D shape based at least on the set of vector stroke paths.

18 . A system comprising:
one or more processors; and one or more memory components storing instructions configured to, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving a training drawing and a representation of a corresponding training three-dimensional (3D) shape into a machine learning model; and
training the machine learning model based on the training drawing and the corresponding training 3D shape to generate a trained machine learning model that generates a two-dimensional (2D) stylized stroke drawing of an input 3D shape based at least on a style of the training drawing and surface geometry features of the input 3D sha

Assignees

• Adobe Inc
• Univ Massachusetts

Inventors

• Hertzmann Aaron
• Fisher Matthew
• Liu Difan
• Kalogerakis Evangelos

Classifications

• G06T2200/04

  involving 3D image data · CPC title

• G06T11/203

  Physics · mapped topic

• G06N3/045

  Combinations of networks · CPC title

• G06T11/001 Primary

  Physics · mapped topic

• G06T11/23

  using straight lines or curves · CPC title