Inpainting digital images using a hybrid wire removal pipeline

Patent metadata

Field	Value
Publication number	US-12524853-B2
Application number	US-202318179855-A
Country	US
Kind code	B2
Filing date	Mar 7, 2023
Priority date	Mar 7, 2023
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for inpainting a digital image using a hybrid wire removal pipeline. For example, the disclosed systems use a hybrid wire removal pipeline that integrates multiple machine learning models, such as a wire segmentation model, a hole separation model, a mask dilation model, a patch-based inpainting model, and a deep inpainting model. Using the hybrid wire removal pipeline, in some embodiments, the disclosed systems generate a wire segmentation from a digital image depicting one or more wires. The disclosed systems also utilize the hybrid wire removal pipeline to extract or identify portions of the wire segmentation that indicate specific wires or portions of wires. In certain embodiments, the disclosed systems further inpaint pixels of the digital image corresponding to the wires indicated by the wire segmentation mask using the patch-based inpainting model and/or the deep inpainting model.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
generating, utilizing a scene semantic segmentation model, a digital image segmentation map from a digital image; generating, utilizing a wire segmentation model, a wire segmentation mask indicating wires depicted within the digital image; generating a hole selection mask by combining the digital image segmentation map and the wire segmentation mask; extracting a first wire hole from a first portion of the hole selection mask and a second wire hole from a second portion of the hole selection mask; dilating the first portion and the second portion of the hole selection mask according to respective wire diameters depicted by pixels in the first portion and the second portion; generating a modified digital image by inpainting the first wire hole and the second wire hole using one or more inpainting models based on dilating the first portion and the second portion of the hole selection mask; and providing the modified digital image for display on a client device.

2 . The computer-implemented method of claim 1 , wherein:
extracting the first wire hole comprises extracting a portion of the hole selection mask corresponding to pixels depicting natural objects; and extracting the second wire hole comprises extracting a portion of the hole selection mask corresponding to pixels depicting manmade objects.

3 . The computer-implemented method of claim 1 , wherein generating the modified digital image comprises:
inpainting the first wire hole using a patch-based inpainting model; and inpainting the second wire hole using a deep inpainting model.

4 . The computer-implemented method of claim 1 , wherein dilating the first portion and the second portion of the hole selection mask comprises:
dilating the first portion of the hole selection mask by a first dilation amount according to a first diameter of a wire depicted by pixels corresponding to the first portion; and dilating the second portion of the hole selection mask by a second dilation amount according to a second diameter of a wire depicted by pixels corresponding to the second portion.

5 . The computer-implemented method of claim 1 , wherein generating the modified digital image comprises using a parallel batching technique to inpaint multiple wire holes in parallel using the one or more inpainting models.

6 . The computer-implemented method of claim 1 , wherein extracting the first wire hole and the second wire hole from the wire segmentation mask further comprises:
generating an inverted digital image segmentation map indicating pixel segments of the digital image; combining the digital image segmentation map with the wire segmentation mask to extract the first wire hole; and combining the inverted digital image segmentation map with the wire segmentation mask to extract the second wire hole.

7 . The computer-implemented method of claim 1 , wherein extracting the first wire hole and the second wire hole from the hole selection mask comprises:
extracting the first wire hole from the first portion of the hole selection mask corresponding to pixels depicting a first length of a wire against a natural background; and extracting the second wire hole from the second portion of the hole selection mask corresponding to pixels depicting a second length of the wire against a manmade background.

8 . A system comprising:
one or more memory devices; and one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
generating, utilizing a scene semantic segmentation model, a digital image segmentation map from a digital image;
generating, utilizing a wire segmentation model, a wire segmentation mask indicating wires depicted within the digital image;
generating a hole selection mask by combining the digital image segmentation map and the wire segmentation mask;
extracting one or more wire holes from the hole selection mask;
determining tile batches for pixels depicting the one or more wire holes of the wire segmentation mask; and
generating a modified digital image by inpainting the one or more wire holes using one or more inpainting models according to the tile batches.

9 . The system of claim 8 , wherein determining the tile batches comprises:
generating a tile grid dividing the digital image into pixel tiles having a set resolution; identifying wire tiles depicting wire pixels from among the pixel tiles of the tile grid; and batching the wire tiles into groups such that wire tiles in a common group do not overlap or overlap by less than a threshold number of pixels.

10 . The system of claim 8 , wherein generating the modified digital image comprises:
inpainting pixels depicting a first portion of a wire against a first background using a patch-based inpainting model; and inpainting pixels depicting a second portion of the wire against a second background using a deep inpainting model.

11 . The system of claim 8 , wherein extracting the one or more wire holes from the hole selection mask comprises:
dilating a first portion of the hole selection mask by a first dilation amount; and dilating a second portion of the hole selection mask by a second dilation amount.

12 . The system of claim 8 , wherein generating the modified digital image comprises inpainting two or more tiles from a tile batch of the tile batches in parallel using the one or more inpainting models.

13 . The system of claim 8 , wherein extracting the one or more wire holes from the hole selection mask comprises:
generating an inverted digital image segmentation map indicating pixel segments of the digital image; combining the digital image segmentation map with the wire segmentation mask to extract a first wire hole; and combining the inverted digital image segmentation map with the wire segmentation mask to extract a second wire hole.

14 . The system of claim 8 , wherein generating the modified digital image comprises:
generating an intermediate digital image by inpainting a first wire hole using a first inpainting model; and inpainting a second wire hole within the intermediate digital image utilizing a second inpainting model.

15 . A non-transitory computer readable medium storing executable instructions which, when executed by at least one processor, cause the at least one processor to perform operations comprising:
generating, utilizing a scene semantic segmentation model, a digital image segmentation map from a digital image; generating, utilizing a wire segmentation model, a wire segmentation mask indicating wires depicted within the digital image; generating a hole selection mask by combining the digital image segmentation map and the wire segmentation mask; extracting a first wire hole from a first portion of the hole selection mask and a second wire hole from a second portion of the hole selection mask; generating a modified digital image by inpainting the first wire hole using a first inpainting model and inpainting the second wire hole using a second inpainting model; and providing the modified digital image for display on a client device.

16 . The non-transitory computer readable medium of claim 15 , wherein generating the modified digital image comprises:
inpainting the first wire hole corresponding to pixels depicting a wire against a first background satisfying a uniformity threshold using a patch-based inpainting model; and inpainting pixels depicting a second portion of the wire against a second background not satisfying the uniformity threshold using a deep

Assignees

• Adobe Inc

Inventors

• Zhou Yuqian
• Barnes Connelly
• Wei Zijun
• Lin Zhe
• Shechtman Elya
• Amirghodsi Sohrab
• Liu Xiaoyang

Classifications

• G06V20/176 Primary

  Urban or other man-made structures · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2207/30184

  Infrastructure · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T5/77 Primary

  Retouching; Inpainting; Scratch removal · CPC title