Generating neural network based perceptual artifact segmentations in modified portions of a digital image

Patent metadata

Field	Value
Publication number	US-12499520-B2
Application number	US-202217815409-A
Country	US
Kind code	B2
Filing date	Jul 27, 2022
Priority date	Jul 27, 2022
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

Methods, systems, and non-transitory computer readable storage media are disclosed for generating neural network based perceptual artifact segmentations in synthetic digital image content. The disclosed system utilizing neural networks to detect perceptual artifacts in digital images in connection with generating or modifying digital images. The disclosed system determines a digital image including one or more synthetically modified portions. The disclosed system utilizes an artifact segmentation machine-learning model to detect perceptual artifacts in the synthetically modified portion(s). The artifact segmentation machine-learning model is trained to detect perceptual artifacts based on labeled artifact regions of synthetic training digital images. Additionally, the disclosed system utilizes the artifact segmentation machine-learning model in an iterative inpainting process. The disclosed system utilizes one or more digital image inpainting models to inpaint in a digital image. The disclosed system utilizes the artifact segmentation machine-learning model detect perceptual artifacts in the inpainted portions for additional inpainting iterations.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
determining, by at least one processor, one or more portions of a digital image to fill by generating one or more synthetically modified portions of the digital image utilizing a digital image inpainting model; and generating one or more artifact segmentations from the digital image by:
determining, utilizing an artifact segmentation machine-learning model, one or more predicted perceptual artifact regions indicating one or more artifacts comprising visible errors in synthetically generated image content, the one or more predicted perceptual artifact regions corresponding to pixels within the one or more synthetically modified portions of the digital image,
wherein the artifact segmentation machine-learning model comprises parameters learned based on labeled artifact regions of synthetic training digital images.

2 . The computer-implemented method of claim 1 , wherein determining the digital image comprises:
generating the one or more synthetically modified portions utilizing an image generation neural network; or selecting the digital image comprising the one or more synthetically modified portions from a database of digital images.

3 . The computer-implemented method of claim 1 , wherein generating the one or more artifact segmentations comprises determining, utilizing the artifact segmentation machine-learning model, a plurality of predicted perceptual artifact regions corresponding to a plurality of separate artifacts within a synthetically modified portion of the digital image.

4 . The computer-implemented method of claim 1 , further comprising:
determining a combined size of one or more artifacts within a synthetically modified portion of the one or more synthetically modified portions of the digital image; determining a size of the synthetically modified portion of the digital image; and generating an artifact ratio metric for the digital image based on the combined size of the one or more artifacts relative to the size of the synthetically modified portion.

5 . The computer-implemented method of claim 4 , further comprising generating, utilizing an image generation neural network, an additional synthetically modified portion replacing an artifact within the synthetically modified portion in response to comparing the artifact ratio metric to a ratio threshold.

6 . The computer-implemented method of claim 4 , further comprising:
determining, based on the artifact ratio metric for the digital image, a first performance of a first image generation neural network utilized to generate the one or more synthetically modified portions of the digital image; determining, based on an additional artifact ratio metric for an additional version of the digital image, a second performance of a second image generation neural network utilized to generate one or more additional synthetically modified portions of the additional version of the digital image; and providing, for display within a graphical user interface, a comparison of the first performance of the first image generation neural network and the second performance of the second image generation neural network.

7 . The computer-implemented method of claim 1 , further comprising:
generating a plurality of candidate digital images comprising synthetically modified portions, the plurality of candidate digital images comprising the digital image; generating a plurality of artifact ratio metrics corresponding to the plurality of candidate digital images based on artifacts relative to sizes of the synthetically modified portions of the plurality of candidate digital images; and selecting the digital image from the plurality of candidate digital images based on the plurality of artifact ratio metrics.

8 . The computer-implemented method of claim 1 , further comprising:
generating predicted artifact bounding regions for portions of the synthetic training digital images; generating modified artifact bounding regions by dilating synthetically modified regions corresponding to the predicted artifact bounding regions by a predetermined amount; and providing, for display at a client device, the synthetic training digital images including the modified artifact bounding regions.

9 . The computer-implemented method of claim 8 , further comprising:
determining a plurality of marked regions of the synthetic training digital images based on user inputs; determining the labeled artifact regions by intersecting the plurality of marked regions and hole masks corresponding to synthetically modified portions of the synthetic training digital images; and learning the parameters of the artifact segmentation machine-learning model based on the labeled artifact regions.

10 . A system comprising:
one or more computer memory devices; and one or more servers configured to cause the system to:
determine one or more portions of a digital image to fill by generating one or more synthetically modified portions of the digital image utilizing a digital image inpainting model;
generate one or more artifact segmentations from the digital image by:
determining, utilizing an artifact segmentation machine-learning model, one or more predicted perceptual artifact regions indicating one or more artifacts comprising visible errors in synthetically generated image content, the one or more predicted perceptual artifact regions corresponding to pixels within the one or more synthetically modified portions of the digital image,
wherein the artifact segmentation machine-learning model comprises parameters learned based on labeled artifact regions of synthetic training digital images; and
generate, for display within a graphical user interface of a client device, one or more indications of the one or more artifact segmentations in the digital image.

11 . The system of claim 10 , wherein the one or more servers are further configured to cause the system to determine the digital image by generating, utilizing the digital image inpainting model, the one or more synthetically modified portions according to a digital image mask associated with a detected object.

12 . The system of claim 10 , wherein the one or more servers are further configured to cause the system to generate the one or more artifact segmentations by:
determining, utilizing the artifact segmentation machine-learning model, a first predicted perceptual artifact region corresponding to a first artifact within a synthetically modified portion of the digital image; and determining, utilizing the artifact segmentation machine-learning model, a second predicted perceptual artifact region corresponding to a second artifact within the synthetically modified portion of the digital image.

13 . The system of claim 10 , wherein the one or more servers are further configured to cause the system to generate the one or more indications of the one or more predicted perceptual artifact regions by:
generating an artifact ratio metric for the digital image based on a combined size of the one or more predicted perceptual artifact regions relative to a combined size of the one or more synthetically modified portions; and providing, for display within a graphical user interface of a client device, an indication to further modify the one or more synthetically modified portions of the digital image in response to comparing the artifact ratio metric to a ratio threshold.

14 . The system of claim 10 , wherein the one or more servers are further configured to cause the system to generate the one or more indications of the one or more predicted perceptual artifact reg

Assignees

• Adobe Inc

Inventors

• Amirghodsi Sohrab
• Zhang Lingzhi
• Lin Zhe
• Shechtman Elya
• Zhou Yuqian
• Barnes Connelly

Classifications

• G06T7/194

  involving foreground-background segmentation · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T5/77 Primary

  Retouching; Inpainting; Scratch removal · CPC title