Mask-robust image inpainting utilizing machine-learning models

Patent metadata

Field	Value
Publication number	US-12536626-B2
Application number	US-202318307546-A
Country	US
Kind code	B2
Filing date	Apr 26, 2023
Priority date	Apr 26, 2023
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for inpainting digital images utilizing mask-robust machine-learning models. In particular, in one or more embodiments, the disclosed systems obtain an initial mask for an object depicted in a digital image. Additionally, in some embodiments, the disclosed systems generate, utilizing a mask-robust inpainting machine-learning model, an inpainted image from the digital image and the initial mask. Moreover, in some implementations, the disclosed systems generate a relaxed mask that expands the initial mask. Furthermore, in some embodiments, the disclosed systems generate a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask.

First claim

What is claimed is:

1 . A system comprising:
one or more memory devices comprising a digital image and an initial mask for an object portrayed in the digital image; and one or more processors coupled to the one or more memory devices that cause the system to train a mask-robust inpainting machine-learning model by:
generating, utilizing the mask-robust inpainting machine-learning model, an inpainted image from the digital image, wherein the inpainted image comprises modified pixels inside and outside the initial mask;
generating, utilizing an additional inpainting machine-learning model, a pseudo-ground-truth inpainted image from the digital image utilizing a dilated mask generated from the initial mask;
determining a measure of loss by comparing the inpainted image and the pseudo-ground-truth inpainted image; and
tuning parameters of the mask-robust inpainting machine-learning model based on the measure of loss.

2 . The system of claim 1 , wherein training the mask-robust inpainting machine-learning model further comprises:
generating a relaxed mask that expands the initial mask; and generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask, wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

3 . The system of claim 2 , wherein the one or more processors further cause the system to:
generate the relaxed mask utilizing a mask refiner machine-learning model; and train the mask refiner machine-learning model by tuning parameters of the mask refiner machine-learning model based on the measure of loss.

4 . The system of claim 1 , wherein training the mask-robust inpainting machine-learning model further comprises:
generating a perturbed mask from the initial mask; generating a relaxed mask that expands the perturbed mask; and generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask, wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

5 . The system of claim 4 , wherein generating the perturbed mask comprises:
replacing the initial mask utilizing a free-form mask; or eroding a selection of boundary pixels of the initial mask.

6 . The system of claim 4 , wherein generating the perturbed mask comprises:
identifying pixel regions within the initial mask; determining probabilities that the pixel regions cover the object portrayed in the digital image; and removing one or more of the pixel regions from the initial mask based on the probabilities.

7 . The system of claim 1 , wherein training the mask-robust inpainting machine-learning model further comprises:
generating, utilizing the mask-robust inpainting machine-learning model, an additional inpainted image from the digital image, wherein the additional inpainted image comprises additional modified pixels inside and outside the initial mask; determining an additional measure of loss by comparing the additional inpainted image and the pseudo-ground-truth inpainted image; and further tuning the parameters of the mask-robust inpainting machine-learning model based on the additional measure of loss.

8 . A computer-implemented method comprising training a mask-robust inpainting machine-learning model by:
generating, utilizing the mask-robust inpainting machine-learning model, an inpainted image from a digital image, wherein the inpainted image comprises modified pixels inside and outside an initial mask for an object portrayed in the digital image; generating, utilizing an additional inpainting machine-learning model, a pseudo-ground-truth inpainted image from the digital image utilizing a dilated mask generated from the initial mask; determining a measure of loss by comparing the inpainted image and the pseudo-ground-truth inpainted image; and tuning parameters of the mask-robust inpainting machine-learning model based on the measure of loss.

9 . The computer-implemented method of claim 8 , wherein training the mask-robust inpainting machine-learning model further comprises:
generating a relaxed mask that expands the initial mask; and generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask, wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

10 . The computer-implemented method of claim 9 , further comprising:
generating the relaxed mask utilizing a mask refiner machine-learning model; and training the mask refiner machine-learning model by tuning parameters of the mask refiner machine-learning model based on the measure of loss.

11 . The computer-implemented method of claim 8 , wherein training the mask-robust inpainting machine-learning model further comprises:
generating a perturbed mask from the initial mask; generating a relaxed mask that expands the perturbed mask; and generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask, wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

12 . The computer-implemented method of claim 11 , wherein generating the perturbed mask comprises:
replacing the initial mask utilizing a free-form mask; or eroding a selection of boundary pixels of the initial mask.

13 . The computer-implemented method of claim 11 , wherein generating the perturbed mask comprises:
identifying pixel regions within the initial mask; determining probabilities that the pixel regions cover the object portrayed in the digital image; and removing one or more of the pixel regions from the initial mask based on the probabilities.

14 . The computer-implemented method of claim 8 , wherein training the mask-robust inpainting machine-learning model further comprises:
generating, utilizing the mask-robust inpainting machine-learning model, an additional inpainted image from the digital image, wherein the additional inpainted image comprises additional modified pixels inside and outside the initial mask; determining an additional measure of loss by comparing the additional inpainted image and the pseudo-ground-truth inpainted image; and further tuning the parameters of the mask-robust inpainting machine-learning model based on the additional measure of loss.

15 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to train a mask-robust inpainting machine-learning model by:
generating, utilizing the mask-robust inpainting machine-learning model, an inpainted image from a digital image, wherein the inpainted image comprises modified pixels inside and outside an initial mask for an object portrayed in the digital image; generating, utilizing an additional inpainting machine-learning model, a pseudo-ground-truth inpainted image from the digital image utilizing a dilated mask generated from the initial mask; determining a measure of loss by comparing the inpainted image and the pseudo-ground-truth inpainted image; and tuning parameters of the mask-robust inpainting machine-learning model based on the measure of loss.

16 . The non-transitory computer-readable medium of claim 15 , wherein traini

Assignees

• Adobe Inc

Inventors

• Amirghodsi Sohrab
• Zhang Lingzhi
• Barnes Connelly
• Shechtman Elya
• Zhou Yuqian
• Lin Zhe

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title