Human inpainting utilizing a segmentation branch for generating an infill segmentation map

Patent metadata

Field	Value
Publication number	US-12347080-B2
Application number	US-202318190556-A
Country	US
Kind code	B2
Filing date	Mar 27, 2023
Priority date	Oct 6, 2022
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.

First claim

What is claimed is:

1. A computer-implemented method comprising:
generating, utilizing a segmentation machine learning model, an initial segmentation map from a digital image to determine a region of a human portrayed within the digital image to inpaint;
generating, utilizing a generative segmentation machine learning model, an infill segmentation map from the digital image and the initial segmentation map by generating one or more human segmentation classifications for the region of the human portrayed within the digital image to inpaint; and
generating, utilizing a human inpainting generative adversarial neural network, a modified digital image from the digital image and the infill segmentation map, wherein the modified digital image comprises modified pixels for the region corresponding to the one or more human segmentation classifications.

2. The computer-implemented method of claim 1 , wherein generating the initial segmentation map comprises generating, utilizing the segmentation machine learning model, an unclassified region corresponding to the region of the human to inpaint in the digital image.

3. The computer-implemented method of claim 2 , further comprising generating the one or more human segmentation classifications for the unclassified region corresponding to the region of the human to inpaint in the digital image.

4. The computer-implemented method of claim 1 , further comprising generating a structural encoding from the infill segmentation map and a visual appearance encoding from the digital image.

5. The computer-implemented method of claim 1 , further comprising generating a structural encoding and a visual appearance encoding further comprises utilizing a hierarchical encoder comprising a plurality of downsampling layers and upsampling layers connected via skip connections.

6. The computer-implemented method of claim 1 , wherein the region of the human to inpaint further comprises a background portion of the digital image and further comprising:
generating, utilizing a background generative adversarial neural network, a modified background portion of the digital image from the digital image; and
generating, utilizing the human inpainting generative adversarial neural network, the modified pixels of the region of the human portrayed within the digital image.

7. The computer-implemented method of claim 6 , further comprising:
inpainting a human portion of the human portrayed within the digital image utilizing the human inpainting generative adversarial neural network; and
inpainting the background portion based on the inpainted human portion and utilizing the background generative adversarial neural network.

8. The computer-implemented method of claim 6 , wherein generating the modified digital image further comprises:
generating an intermediate digital image by removing the region of the human and inpainting the background portion utilizing the background generative adversarial neural network;
generating, utilizing the human inpainting generative adversarial neural network, the modified pixels for the region corresponding to the one or more human segmentation classifications; and
generating the modified digital image by inserting the modified pixels into the intermediate digital image.

9. A system comprising:
one or more memory devices comprising a digital image, a generative segmentation machine learning model, a human inpainting generative adversarial neural network, and a background generative adversarial neural network; and
one or more processors configured to cause the system to:
generate, utilizing a segmentation machine learning model, an initial segmentation map from the digital image to determine a region of the digital image to inpaint, the region comprising a human portion of the digital image and a background portion of the digital image;

generate, utilizing the generative segmentation machine learning model, an infill segmentation map from the digital image and the initial segmentation map by generating a human segmentation classification for the human portion;
generate, utilizing the human inpainting generative adversarial neural network, a modified human portion of the digital image from the digital image and the infill segmentation map; and
generate, utilizing the background generative adversarial neural network, a modified background portion of the digital image from the digital image.

10. The system of claim 9 , wherein the one or more processors are configured to cause the system to generate a modified digital image by combining the modified background portion and the modified human portion of the digital image from the digital image.

11. The system of claim 10 , wherein the one or more processors are configured to cause the system to:
generate, utilizing an encoder, a structural encoding from the infill segmentation map;
generate, utilizing the encoder, a visual appearance encoding based on the human portion of the digital image; and
generate the modified digital image from the modified background portion, the structural encoding, and the visual appearance encoding.

12. The system of claim 9 , wherein the one or more processors are configured to cause the system to generate a mask for the region of the human portion in the digital image and a mask for the background portion of the digital image.

13. The system of claim 12 , wherein the one or more processors are configured to cause the system to generate a modified digital image by:
modifying pixels for the region corresponding to the human portion of the digital image utilizing the mask for the region of the human portion; and
modifying pixels for the region corresponding to the background portion of the digital image utilizing the mask for the background portion.

14. The system of claim 9 , wherein the one or more processors are configured to cause the system to:
generate the initial segmentation map comprising an unclassified region corresponding to the region of the human portion to inpaint; and
generate one or more human segmentation classifications for the unclassified region corresponding to the region of the human portion to inpaint in the digital image.

15. A non-transitory computer-readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
generating, utilizing a segmentation machine learning model, an initial segmentation map from a digital image to determine a region of a human portrayed within the digital image to inpaint;
generating, utilizing a generative segmentation machine learning model, an infill segmentation map from the digital image and the initial segmentation map by generating one or more human segmentation classifications for the region of the human portrayed within the digital image to inpaint;
generating, utilizing a hierarchical encoder comprising a plurality of downsampling layers and upsampling layers connected via skip connections, a structural encoding from the infill segmentation map; and
generating, utilizing a human inpainting generative adversarial neural network, a modified digital image from the digital image and the structural encoding, the modified digital image comprising modified pixels for the region corresponding to the one or more human segmentation classifications.

16. The non-transitory computer-readable medium of claim 15 , further comprising:
generating, utilizing the hierarchical encoder, a visual appearance encoding based on the digital image; and
generating, utilizing the human inpainting generative adversarial neural networ

Assignees

• Adobe Inc

Inventors

• Singh Krishna Kumar
• Li Yijun
• Lu Jingwan
• Aksit Duygu Ceylan
• Wang Yangtuanfeng
• Yang Jimei
• Hinz Tobias
• Liu Qing
• Zhang Jianming
• Lin Zhe

Classifications

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2207/30196

  Human being; Person · CPC title