Portrait stylization framework using a two-path image stylization and blending

Patent metadata

Field	Value
Publication number	US-11954828-B2
Application number	US-202117501990-A
Country	US
Kind code	B2
Filing date	Oct 14, 2021
Priority date	Oct 14, 2021
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

Systems and method directed to generating a stylized image are disclosed. In particular, the method includes, in a first data path, (a) applying first stylization to an input image and (b) applying enlargement to the stylized image from (a). The method also includes, in a second data path, (c) applying segmentation to the input image to identify a face region of the input image and generate a mask image, and (d) applying second stylization to an entirety of the input image and inpainting to the identified face region of the stylized image. Machine-assisted blending is performed based on (1) the stylized image after the enlargement from the first data path, (2) the inpainted image from the second data path, and (3) the mask image, in order to obtain a final stylized image.

First claim

What is claimed is:

1. A method for generating a stylized image, the method comprising:
in a first data path,
applying first stylization to an input image to generate a first stylized image, and
applying enlargement to the first stylized image to generate an enlarged stylized image;
in a second data path,
applying segmentation to the input image to identify a face region of the input image and generate a mask image,
applying second stylization to an entirety of the input image and inpainting based on the identified face region to generate an inpainted image; and
performing machine-assisted blending based on the enlarged stylized image from the first data path, the inpainted image from the second data path, and the mask image to obtain a final stylized image.

2. The method of claim 1 , further comprising:
in the second data path,
partially inserting the face region from the input image into the inpainted image to generate a partially inserted image, and
performing color transfer between the input image and the partially inserted image.

3. The method of claim 1 , further comprising:
in the first data path,
prior to applying the first stylization, normalizing the input image,
after applying the enlargement, applying segmentation to the enlarged stylized image to identify a face region of the enlarged stylized image, and
aligning the identified face region of the enlarged stylized image with the input image before the normalizing.

4. The method of claim 1 , wherein the machine-assisted blending is Laplacian pyramid blending.

5. The method of claim 1 , wherein the first stylization and the second stylization are performed using at least one trained generative adversarial network (GAN) generator.

6. The method of claim 5 , wherein the trained GAN generator comprises an AgileGAN generator or a StyleGAN2 generator.

7. The method of claim 1 , wherein the first stylization is different from the second stylization.

8. The method of claim 1 , wherein the first data path and the second data path are implemented simultaneously.

9. The method of claim 1 , wherein the first data path is implemented before or after the second data path.

10. A system configured to generate a stylized image, the system comprising:
a processor; and
memory including instructions, which when executed by the processor, causes the processor to:
in a first data path,
apply first stylization to an input image to generate a first stylized image, and
apply enlargement to the first stylized image to generate an enlarged stylized image;
in a second data path,
apply segmentation to the input image to identify a face region of the input image and generate a mask image, and
apply second stylization to an entirety of the input image and inpainting based on the identified face region to generate an inpainted image; and
perform machine-assisted blending based on the enlarged stylized image from the first data path, the inpainted image from the second data path, and the mask image to obtain a final stylized image.

11. The system of claim 10 , wherein the instructions, when executed by the processor, cause the processor to:
in the second data path,
partially inserting the face region from the input image into the inpainted image to generate a partially inserted image, and
performing color transfer between the input image and the partially inserted image.

12. The system of claim 10 , wherein the instructions, when executed by the processor, cause the processor to:
in the first data path,
prior to applying the first stylization, normalize the input image,
after applying the enlargement, apply segmentation to the enlarged stylized image to identify a face region of the enlarged stylized image, and
align the identified face region of the enlarged stylized image with the input image before the normalizing.

13. The system of claim 10 , wherein the machine-assisted blending is Laplacian pyramid blending.

14. The system of claim 10 , wherein the first stylization and the second stylization are performed using at least one trained generative adversarial network (GAN) generator.

15. The system of claim 14 , wherein the trained GAN generator comprises an AgileGAN generator or a StyleGAN2 generator.

16. The system of claim 10 , wherein the first stylization is different from the second stylization.

17. The system of claim 10 , wherein the first data path and the second data path are implemented simultaneously.

18. The system of claim 10 , wherein the first data path is implemented before or after the second data path.

19. A non-transitory computer-readable storage medium including instructions, which when executed by a processor, cause the processor to:
in a first data path,
apply first stylization to an input image to generate a first stylized image, and
apply enlargement to the first stylized image to generate an enlarged stylized image;
in a second data path,
apply segmentation to the input image to identify a face region of the input image and generate a mask image, and
apply second stylization to an entirety of the input image and inpainting based on the identified face region to generate an inpainted image; and
perform machine-assisted blending based on the enlarged stylized image from the first data path, the inpainted image from the second data path, and the mask image to obtain a final stylized image.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the instructions, which when executed by the processor, cause the processor to:
in the second data path,
partially inserting the face region from the input image into the inpainted image to generate a partially inserted image, and
performing color transfer between the input image and the partially inserted image; and
in the first data path,
prior to applying the first stylization, normalize the input image,
after applying the enlargement, apply segmentation to the enlarged stylized image to identify a face region of the enlarged stylized image, and
align the identified face region of the enlarged stylized image with the input image before the normalizing.

Assignees

• Lemon Inc

Inventors

• Liu Jing
• Lai Chunpong
• Song Guoxian
• Luo Linjie
• Yuan Ye

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T5/005 Primary

  Physics · mapped topic

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T7/11

  Region-based segmentation · CPC title