Improving digital image inpainting utilizing plane panoptic segmentation and plane grouping

Patent metadata

Field	Value
Publication number	US-12437375-B2
Application number	US-202418743497-A
Country	US
Kind code	B2
Filing date	Jun 14, 2024
Priority date	Nov 5, 2021
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately generating inpainted digital images utilizing a guided inpainting model guided by both plane panoptic segmentation and plane grouping. For example, the disclosed systems utilize a guided inpainting model to fill holes of missing pixels of a digital image as informed or guided by an appearance guide and a geometric guide. Specifically, the disclosed systems generate an appearance guide utilizing plane panoptic segmentation and generate a geometric guide by grouping plane panoptic segments. In some embodiments, the disclosed systems generate a modified digital image by implementing an inpainting model guided by both the appearance guide (e.g., a plane panoptic segmentation map) and the geometric guide (e.g., a plane grouping map).

First claim

What is claimed is:

1. A method comprising:
generating, using an inpainting model, a preliminary inpainted image by inpainting one or more holes of an initial digital image;
generating, from the preliminary inpainted image, an appearance guide from the preliminary inpainted image by:
detecting surface planes within the preliminary inpainted image utilizing a plane detection model; and
detecting panoptic segments within the preliminary inpainted image utilizing a panoptic segmentation model that distinguishes between instances of objects with a shared semantic label; and

generating a final inpainted image by utilizing a guided inpainting model informed by the appearance guide as a guide for inpainting the one or more holes of the initial digital image.

2. The method of claim 1 , wherein generating the appearance guide further comprises:
grouping plane panoptic segments of the appearance guide into a plane grouping map by:
determining normal vectors for plane panoptic segments made up of the panoptic segments and the surface planes; and
grouping the plane panoptic segments into surface plane groups according to the normal vectors.

3. The method of claim 1 , wherein generating the final inpainted image comprises utilizing the guided inpainting model to inpaint the one or more holes of the initial digital image according to an appearance guidance optimization algorithm that constrains sampling based on structure separation indicated by the appearance guide.

4. The method of claim 1 , further comprising generating a geometric guide by grouping the surface planes of the appearance guide based on comparing respective normal vectors of the surface planes;
wherein generating the final inpainted image comprises utilizing the guided inpainting model informed by the geometric guide.

5. The method of claim 4 , wherein generating the final inpainted image comprises utilizing the guided inpainting model informed by the appearance guide and the geometric guide to guide inpainting the one or more holes of the initial digital image.

6. The method of claim 1 , wherein generating the appearance guide comprises generating, for a pixel of the preliminary inpainted image, a triplet label comprising a semantic label, an instance label, and a surface plane identification for content depicted by the pixel.

7. The method of claim 1 , further comprising providing the final inpainted image for display on a client device.

8. A system comprising:
one or more memory components; and
one or more processors coupled to the one or more memory components, wherein the one or more processors are configured to cause the system to perform operations comprising:
generating, using an inpainting model, a preliminary inpainted image by inpainting one or more holes of an initial digital image;
generating, from the preliminary inpainted image, an appearance guide that indicates semantic classes and surface planes from the preliminary inpainted image by:
detecting surface planes within the preliminary inpainted image utilizing a plane detection model; and
detecting panoptic segments within the preliminary inpainted image utilizing a panoptic segmentation model that distinguishes between instances of objects with a shared semantic label; and

generating a final inpainted image by utilizing a guided patch match model informed by the appearance guide for inpainting the one or more holes of the initial digital image.

9. The system of claim 8 , further comprising generating a geometric guide by grouping the surface planes of the appearance guide based on comparing respective normal vectors of the surface planes;
wherein generating the final inpainted image comprises utilizing the guided patch match model informed by the geometric guide.

10. The system of claim 9 , wherein the operations further comprise grouping plane panoptic segments of the appearance guide into a plane grouping map by:
determining normal vectors for plane panoptic segments made up of the panoptic segments and the surface planes; and
grouping the plane panoptic segments into surface plane groups according to the normal vectors.

11. The system of claim 10 , wherein generating the final inpainted image comprises utilizing the guided patch match model to inpaint the one or more holes of the initial digital image as guided by the appearance guide and the plane grouping map.

12. The system of claim 8 , wherein generating the final inpainted image comprises utilizing the guided patch match model to inpaint the one or more holes of the initial digital image according to an appearance guidance optimization algorithm that constrains sampling based on structure separation indicated by the appearance guide.

13. The system of claim 8 , wherein generating the appearance guide comprises generating, for pixels of the preliminary inpainted image, triplet labels that each include a semantic label, an instance label, and a surface plane identification for content depicted by the pixels.

14. The system of claim 8 , wherein:
generating the preliminary inpainted image is at a first resolution; and
generating the final inpainted image is at a second resolution higher than the first resolution.

15. A non-transitory computer readable medium storing instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
generating, using an inpainting model, a preliminary inpainted image by inpainting one or more holes of an initial digital image;
generating, from the preliminary inpainted image, a plane panoptic segmentation map that combines panoptic segments and surface planes from the preliminary inpainted image;
determining a plane grouping map by grouping the surface planes of the plane panoptic segmentation map based on comparing respective normal vectors of the surface planes; and
generating a final inpainted image by utilizing a guided inpainting model informed by the plane panoptic segmentation map and the plane grouping map to guide inpainting the one or more holes of the initial digital image.

16. The non-transitory computer readable medium of claim 15 , wherein generating the plane panoptic segmentation map comprises generating, for a pixel of the preliminary inpainted image, a triplet label comprising a semantic label, an instance label, and a surface plane identification for content depicted by the pixel.

17. The non-transitory computer readable medium of claim 15 , wherein determining the plane grouping map comprises:
performing an early grouping process comprising detecting and clustering lines depicted across all of the preliminary inpainted image according to vanishing points; and
performing a late grouping process comprising detecting lines depicted in individual plane panoptic segments of the plane panoptic segmentation map.

18. The non-transitory computer readable medium of claim 15 , wherein generating the plane panoptic segmentation map comprises:
detecting the surface planes within the preliminary inpainted image utilizing a plane detection model;
detecting the panoptic segments within the preliminary inpainted image utilizing a panoptic segmentation model that distinguishes between instances of objects with a shared semantic label; and
combining the surface planes and the panoptic segments into shared labels.

19. The non-transitory computer readable medium of claim 15 , wherein determining the plane grouping map comprises:
determining normal vectors for the surface planes;
comparing the normal vectors to determine differences a

Assignees

• Adobe Inc

Inventors

• Zhou Yuqian
• Barnes Connelly
• Amirghodsi Sohrab
• Shechtman Elya

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06N3/02

  Neural networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11

  Region-based segmentation · CPC title