Generating stylized-stroke images from source images utilizing style-transfer-neural networks with non-photorealistic-rendering

Patent metadata

Field	Value
Publication number	US-10748324-B2
Application number	US-201816184289-A
Country	US
Kind code	B2
Filing date	Nov 8, 2018
Priority date	Nov 8, 2018
Publication date	Aug 18, 2020
Grant date	Aug 18, 2020

Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that integrate (or embed) a non-photorealistic rendering (“NPR”) generator with a style-transfer-neural network to generate stylized images that both correspond to a source image and resemble a stroke style. By integrating an NPR generator with a style-transfer-neural network, the disclosed methods, non-transitory computer readable media, and systems can accurately capture a stroke style resembling one or both of stylized edges or stylized shadings. When training such a style-transfer-neural network, the integrated NPR generator can enable the disclosed methods, non-transitory computer readable media, and systems to use real-stroke drawings (instead of conventional paired-ground-truth drawings) for training the network to accurately portray a stroke style. In some implementations, the disclosed methods, non-transitory computer readable media, and systems can either train or apply a style-transfer-neural network that captures a variety of stroke styles, such as different edge-stroke styles or shading-stroke styles.

First claim

We claim:

1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:
generate, utilizing a non-photorealistic-rendering (“NPR”) generator corresponding to a style-transfer-neural network, a simplified image of a source image, the simplified image depicting NPR features extracted from the source image and the source image exhibiting a stroke style;
extract, utilizing an encoder of the style-transfer-neural network, a feature map from the simplified image depicting NPR features extracted from the source image; and
generate, utilizing a decoder of the style-transfer-neural network, a stylized-stroke image corresponding to the source image and exhibiting the stroke style by decoding the feature map.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
receive an indication of a user selection of a stroke-style setting corresponding to the stroke style from among a first stroke-style setting and a second stroke-style setting; and
based on receiving the indication of the user selection of the stroke-style setting, generate the stylized-stroke image exhibiting the stroke style.

3. The non-transitory computer readable medium of claim 1 , wherein the stoke style comprises one of pencil, woodcut, ink, crayon, or charcoal.

4. The non-transitory computer readable medium of claim 1 , wherein the instructions, when executed by the at least one processor, cause the computer system to:
receive an indication of a user selection of a shading-style setting corresponding to a shading-stroke style from among a first shading-style setting and a second shading-style setting; and
generate the stylized-stroke image exhibiting the stroke style by generating a stylized-shading image comprising shading exhibiting the shading-stroke style based on receiving the indication of the user selection of the shading-style setting.

5. The non-transitory computer readable medium of claim 4 , wherein the shading-stroke style comprise one of lines, crossed, smudge, or stippling.

6. The non-transitory computer readable medium of claim 1 , wherein:
the style-transfer-neural network comprises an edge-style-transfer-neural network trained to generate stylized-edge images exhibiting an edge-stroke style;
the decoder of the style-transfer-neural network comprises a decoder of the edge-style-transfer-neural network;
the stroke style comprises the edge-stroke style; and
the instructions, when executed by the at least one processor, cause the computer system to generate the stylized-stroke image by generating, utilizing the decoder of the edge-style-transfer-neural network, a stylized-edge image corresponding to the source image, the stylized-edge image comprising edges exhibiting the edge-stroke style.

7. The non-transitory computer readable medium of claim 1 , wherein:
the style-transfer-neural network comprises a shading-style-transfer-neural network trained to generate stylized-shading images exhibiting a shading-stroke style;
the decoder of the style-transfer-neural network comprises a decoder of the shading-style-transfer-neural network;
the stroke style comprises the shading-stroke style; and
the instructions, when executed by the at least one processor, cause the computer system to generate the stylized-stroke image by generating, utilizing the decoder of the shading-style-transfer-neural network, a stylized-shading image corresponding to the source image, the stylized-shading image comprising shading exhibiting the shading-stroke style.

8. The non-transitory computer readable medium of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
generate, utilizing an additional NPR generator, an additional simplified image of the source image;
based on the additional simplified image, generate, utilizing an edge-style-transfer-neural network, a stylized-edge image corresponding to the source image and comprising edges exhibiting an edge-stroke style;
generate, utilizing a style-fusion-neural network, a fusion map for synthesizing stroke styles from the stylized-edge image and the stylized-shading image; and
based on the fusion map, generate a style-fusion image comprising the edges exhibiting the edge-stroke style and the shading exhibiting the shading-stroke style.

9. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to generate the simplified image depicting NPR features extracted from the source image by:
generating the simplified image comprising edge depictions from the source image utilizing an extended difference-of-gaussians operator; or
generating the simplified image comprising a contrast abstraction from the source image utilizing an objective-abstraction function.

10. The non-transitory computer readable medium of claim 1 , wherein the source image comprises a natural photograph.

11. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to train the style-transfer-neural network by:
generating, utilizing the NPR generator, a simplified-training image of a stroke-training image exhibiting the stroke style;
extracting, utilizing the encoder of the style-transfer-neural network, a training-feature map from the simplified-training image;
based on the training-feature map, generating, utilizing the decoder of the style-transfer-neural network, a stylized-stroke-sample image exhibiting the stroke style;
determining an adversarial loss using an adversarial-loss function based on a discriminator-neural network comparing the stylized-stroke-sample image and a real-stroke drawing;
determining a reconstruction loss from a reconstruction-loss function based on a comparison of the stylized-stroke-sample image and the stroke-training image; and
adjusting network parameters of the style-transfer-neural network based on the determined adversarial loss and the determined reconstruction loss.

12. The non-transitory computer readable medium of claim 11 , wherein:
the style-transfer-neural network comprises an edge-style-transfer-neural network, the decoder of the style-transfer-neural network comprises a decoder of the edge-style-transfer-neural network, the stroke style comprises an edge-stroke style, and the stroke-training image comprises an edge-stroke-training image; and
the instructions, when executed by the at least one processor, cause the computer system to:
generate the simplified-training image of the stroke-training image by generating, utilizing the NPR generator, the simplified-training image comprising edge depictions from the edge-stroke-training image; and
generate the stylized-stroke-sample image by generating, utilizing the decoder of the edge-style-transfer-neural network, a stylized-edge-sample image exhibiting the edge-stroke style.

13. The non-transitory computer readable medium of claim 11 , wherein:
the style-transfer-neural network comprises a shading-style-transfer-neural network, the decoder of the style-transfer-neural network comprises a decoder of the shading-style-transfer-neural network, the stroke style comprises a shading-stroke style, and the stroke-training image comprises a shading-stroke-training image; and
the instructions, when executed by the at least one processor, cause the computer system to:
generate the simplified-training

Assignees

• Adobe Inc

Inventors

• Shechtman Elya
• Li Yijun
• Fang Chen
• Hertzmann Aaron

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/0475

  Generative networks · CPC title