Panoptically guided inpainting utilizing a panoptic inpainting neural network

Patent metadata

Field	Value
Publication number	US-12505518-B2
Application number	US-202217937695-A
Country	US
Kind code	B2
Filing date	Oct 3, 2022
Priority date	Oct 3, 2022
Publication date	Dec 23, 2025
Grant date	Dec 23, 2025

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for panoptically guiding digital image inpainting utilizing a panoptic inpainting neural network. In some embodiments, the disclosed systems utilize a panoptic inpainting neural network to generate an inpainted digital image according to panoptic segmentation map that defines pixel regions corresponding to different panoptic labels. In some cases, the disclosed systems train a neural network utilizing a semantic discriminator that facilitates generation of digital images that are realistic while also conforming to a semantic segmentation. The disclosed systems generate and provide a panoptic inpainting interface to facilitate user interaction for inpainting digital images. In certain embodiments, the disclosed systems iteratively update an inpainted digital image based on changes to a panoptic segmentation map.

First claim

What is claimed is:

1 . A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
identifying, for a digital image, a binary mask indicating a designated area of pixels to be replaced; generating an intermediate digital image by preliminarily inpainting the designated area of pixels to be replaced within the digital image; determining, for the intermediate digital image, a panoptic segmentation map comprising panoptic labels for regions of the intermediate digital image, including one or more panoptic labels corresponding to the designated area of pixels to be replaced within the digital image; and generating from the digital image and the panoptic segmentation map, utilizing a panoptic inpainting neural network, an inpainted digital image depicting replacement pixels for the designated area of pixels within the digital image according to the one or more panoptic labels of the designated area of pixels.

2 . The non-transitory computer readable medium of claim 1 , wherein determining the panoptic segmentation map comprises determining different panoptic labels for regions of the intermediate digital image sharing a common semantic label.

3 . The non-transitory computer readable medium of claim 1 , wherein generating the inpainted digital image comprises:
utilizing the panoptic inpainting neural network to inpaint a first portion of the designated area of the digital image with pixels that correspond to a first panoptic label; and utilizing the panoptic inpainting neural network to inpaint a second portion of the designated area of the digital image with pixels that correspond to a second panoptic label.

4 . The non-transitory computer readable medium of claim 1 , wherein generating the inpainted digital image comprises utilizing the panoptic inpainting neural network to inpaint the designated area of the digital image by filling missing pixels of the designated area.

5 . The non-transitory computer readable medium of claim 1 , wherein determining the panoptic segmentation map comprises utilizing a segmentation neural network to generate the panoptic labels for the regions of the intermediate digital image.

6 . The non-transitory computer readable medium of claim 1 , wherein identifying the binary mask comprises utilizing a mask generator neural network to determine the designated area from the digital image.

7 . The non-transitory computer readable medium of claim 1 , wherein generating the inpainted digital image comprises utilizing the panoptic inpainting neural network to inpaint the designated area of the digital image with the replacement pixels according to the panoptic segmentation map and the binary mask.

8 . A system comprising:
one or more memory devices comprising a digital image depicting pixels to be replaced and a panoptic inpainting neural network; and one or more processors configured to cause the system to generate an inpainted digital image utilizing the panoptic inpainting neural network by:
identifying, for a digital image, a binary mask indicating a designated area of pixels to be replaced;
generating an intermediate digital image by preliminarily inpainting the designated area of pixels to be replaced within the digital image;
determining, for the intermediate digital image, a panoptic segmentation map comprising panoptic labels for regions of the intermediate digital image, including one or more panoptic labels corresponding to the designated area of pixels to be replaced within the digital image; and
generating from the digital image, the panoptic segmentation map, and the binary mask, utilizing the panoptic inpainting neural network, the inpainted digital image depicting replacement pixels for the designated area of pixels within the digital image according to the one or more panoptic labels of the designated area of pixels.

9 . The system of claim 8 , wherein determining the panoptic segmentation map comprises:
determining boundaries between regions of the intermediate digital image corresponding to different semantic labels; and determining boundaries between regions of the intermediate digital image corresponding to different instances of shared semantic labels.

10 . The system of claim 8 , wherein generating the inpainted digital image comprises utilizing the panoptic inpainting neural network to inpaint the designated area of the digital image by filling the designated area with pixels corresponding to panoptic labels of objects depicted within the digital image.

11 . The system of claim 8 , wherein determining the panoptic segmentation map comprises utilizing a segmentation neural network to generate the panoptic labels for the panoptic segmentation map from the intermediate digital image.

12 . The system of claim 8 , wherein identifying the binary mask comprises utilizing a mask generator neural network to determine the designated area from the digital image.

13 . The system of claim 8 , wherein the one or more processors are further configured to generate the inpainted digital image utilizing the panoptic inpainting neural network by receiving an indication of user interaction from a client device modifying the panoptic segmentation map.

14 . The system of claim 13 , wherein generating the inpainted digital image comprises utilizing the panoptic inpainting neural network to inpaint the designated area of the digital image according to the panoptic segmentation map modified via the client device.

15 . A computer-implemented method comprising:
identifying, for a digital image, a binary mask indicating a designated area of pixels to be replaced; generating an intermediate digital image by preliminarily inpainting the designated area of pixels to be replaced within the digital image; determining, for the intermediate digital image, a panoptic segmentation map comprising panoptic labels for regions of the intermediate digital image, including one or more panoptic labels corresponding to the designated area of pixels to be replaced within the digital image; and generating from the digital image and the panoptic segmentation map, utilizing a panoptic inpainting neural network, an inpainted digital image depicting replacement pixels for the designated area of pixels within the digital image according to the one or more panoptic labels of the designated area of pixels.

16 . The computer-implemented method of claim 15 , wherein determining the panoptic segmentation map comprises receiving an indication of user interaction from a client device defining the panoptic labels for the regions of the digital image.

17 . The computer-implemented method of claim 15 , wherein generating the inpainted digital image comprises utilizing the panoptic inpainting neural network to inpaint the designated area of the digital image by filling the designated area with pixels corresponding to a panoptic label of an object not depicted within the digital image.

18 . The computer-implemented method of claim 17 , further comprising receiving an indication of user interaction from a client device to modify the panoptic segmentation map to include the panoptic label of the object not depicted within the digital image.

19 . The computer-implemented method of claim 15 , wherein identifying the binary mask comprises:
receiving, from a client device, an indication of a designated area of pixels to be replaced; and generating the binary mask indicating the designated area

Assignees

• Adobe Inc

Inventors

• Lin Zhe
• Zheng Haitian
• Shechtman Elya
• Zhang Jianming
• Lu Jingwan
• Xu Ning
• Liu Qing
• Cohen Scott
• Amirghodsi Sohrab

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/20092

  Interactive image processing based on input by user · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title