Object class inpainting in digital images utilizing class-specific inpainting neural networks

Patent metadata

Field	Value
Publication number	US-12430725-B2
Application number	US-202217663317-A
Country	US
Kind code	B2
Filing date	May 13, 2022
Priority date	May 13, 2022
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing class-specific cascaded modulation inpainting neural network. For example, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate replacement pixels portraying a particular target object class. To illustrate, in response to user selection of a replacement region and target object class, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image that portrays an instance of the target object class within the replacement region. Moreover, in one or more embodiments the disclosed systems train class-specific cascaded modulation inpainting neural networks corresponding to a variety of target object classes, such as a sky object class, a water object class, a ground object class, or a human object class.

First claim

What is claimed is:

1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving, via a user interface of a client device, an indication of a replacement region of a digital image and a target object class;
generating replacement pixels for the replacement region utilizing a class-specific inpainting neural network corresponding to the target object class and having a plurality of cascaded modulation layers, wherein a given cascaded modulation layer comprises a global modulation block and a spatial modulation block, by:
utilizing the global modulation blocks of the plurality of cascaded modulation layers to apply a modulation based on a global feature code to capture global predictions; and
utilizing the spatial modulation blocks of the plurality of cascaded modulation layers to apply a spatial modulation to refine the global predictions; and
providing, for display via the client device, an inpainted digital image comprising the replacement pixels such that the inpainted digital image portrays an instance of the target object class within the replacement region.

2. The non-transitory computer readable medium of claim 1 , wherein receiving the indication of the replacement region and the target object class comprises:
providing, for display via the user interface, the digital image; and
receiving, via the user interface, a user selection corresponding to the replacement region utilizing a selection tool corresponding to the target object class.

3. The non-transitory computer readable medium of claim 2 , further comprising: determining the replacement region utilizing a segmentation model and the user selection.

4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
utilizing the global modulation block to generate a global feature map to apply a modulation based on a global feature code; and
utilizing the spatial modulation block to perform a spatial modulation utilizing the global feature map.

5. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising generating the replacement pixels utilizing the class-specific inpainting neural network corresponding to at least one of: a sky object class, a water object class, a ground object class, or a human object class.

6. The non-transitory computer readable medium of claim 1 , wherein generating the replacement pixels utilizing the class-specific inpainting neural network comprises generating an image encoding utilizing encoder layers of a class-specific cascaded modulation inpainting neural network.

7. The non-transitory computer readable medium of claim 6 , wherein generating the image encoding utilizing the encoder layers of the class-specific cascaded modulation inpainting neural network comprises:
generating positional encodings corresponding to different resolutions of the encoder layers; and
generating a plurality of encoding feature vectors utilizing the encoder layers and the positional encodings.

8. The non-transitory computer readable medium of claim 6 , wherein generating the replacement pixels comprises generating the replacement pixels utilizing cascaded modulation decoder layers of the class-specific cascaded modulation inpainting neural network from the image encoding.

9. A computer-implemented method comprising:
receiving, via a user interface of a client device, a user interaction with a digital image comprising an indication to replace a sky replacement region of the digital image;
determining, utilizing a panoptic segmentation model, a sky target object class based on the indication to replace the sky replacement region of a digital image;
selecting, based on the indicated sky target object class, a class-specific cascaded modulation inpainting neural network trained to generate sky regions for digital images;
generating sky replacement pixels for the sky replacement region utilizing the class-specific cascaded modulation inpainting neural network trained to generate sky regions for digital images and having a plurality of cascaded modulation layers, wherein a given cascaded modulation layer comprises a global modulation block and a spatial modulation block, by:
utilizing the global modulation blocks of the plurality of cascaded modulation layers to apply a modulation based on a global feature code to capture global predictions; and
utilizing the spatial modulation blocks of the plurality of cascaded modulation layers to apply a spatial modulation to refine the global predictions; and
providing, for display via the client device, an inpainted digital image comprising the sky replacement pixels within the sky replacement region.

10. The computer-implemented method of claim 9 , further comprising determining the sky replacement region based on user input selecting a portion of the digital image.

11. The computer-implemented method of claim 9 , further comprising:
selecting the class-specific cascaded modulation inpainting neural network trained to generate sky regions from a plurality of class-specific cascaded modulation inpainting neural networks based on the indication to replace the sky replacement region.

12. The computer-implemented method of claim 9 , further comprising generating the sky replacement pixels utilizing cascaded modulation decoder layers of the class-specific cascaded modulation inpainting neural network from an image encoding.

13. The computer-implemented method of claim 12 , wherein generating the sky replacement pixels comprises generating positional encodings corresponding to different resolutions of the cascaded modulation decoder layers.

14. The computer-implemented method of claim 13 , further comprising generating the sky replacement pixels utilizing the cascaded modulation decoder layers of the class-specific cascaded modulation inpainting neural network, the image encoding, and the positional encodings.

15. A system comprising:
one or more memory devices; and
one or more processors configured to cause the system to:
receive, via a user interface of a client device, an indication of a replacement region of a digital image and a target object class;

generate replacement pixels for the replacement region utilizing a class-specific inpainting neural network corresponding to the target object class and having a plurality of cascaded modulation layers, wherein a given cascaded modulation layer comprises a global modulation block and a spatial modulation block, by:
utilizing the global modulation blocks of the plurality of cascaded modulation layers to apply a modulation based on a global feature code to capture global predictions; and
utilizing the spatial modulation blocks of the plurality of cascaded modulation layers to apply a spatial modulation to refine the global predictions;
provide, for display via the client device, an inpainted digital image comprising the replacement pixels such that the inpainted digital image portrays an instance of the target object class within the replacement region.

16. The system of claim 15 , wherein receiving the indication of the replacement region and the target object class comprises:
providing, for display via the user interface, the digital image; and
receiving, via the user interface, a user selection correspond

Assignees

• Adobe Inc

Inventors

• Zheng Haitian
• Lin Zhe
• Lu Jingwan
• Cohen Scott
• Shechtman Elya
• Barnes Connelly
• Zhang Jianming
• Xu Ning
• Amirghodsi Sohrab

Classifications

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T5/77 Primary

  Retouching; Inpainting; Scratch removal · CPC title