System and method for image inpainting

Patent metadata

Field	Value
Publication number	US-11580622-B2
Application number	US-202217694687-A
Country	US
Kind code	B2
Filing date	Mar 15, 2022
Priority date	Mar 15, 2021
Publication date	Feb 14, 2023
Grant date	Feb 14, 2023

Abstract

A system for image inpainting is provided, including an encoder, a decoder, and a sketch tensor space of a third-order tensor; wherein the encoder includes an improved wireframe parser and a canny detector, and a pyramid structure sub-encoder; the improved wireframe parser is used to extract line maps from an original image input to the encoder, the canny detector is used to extract edge maps from the original image, and the pyramid structure sub-encoder is used to generate the sketch tensor space based on the original image, the line maps and the edge maps; and the decoder outputs an inpainted image from the sketch tensor space. A method thereof is also provided.

First claim

The invention claimed is:

1. A system for image inpainting, comprising an encoder, a decoder, and a sketch tensor space of a third-order tensor; wherein:
the encoder includes a wireframe parser and a canny detector, and a pyramid structure sub-encoder;
the wireframe parser is used to extract line maps from an original image input to the encoder, the canny detector is used to extract edge maps from the original image, and the pyramid structure sub-encoder is used to generate the sketch tensor space based on the original image, the line maps and the edge maps; and
the decoder outputs an inpainted image from the sketch tensor space.

2. The system of claim 1 , wherein the wireframe parser extracts a junction set and a line set paired by junctions of the junction set.

3. The system of claim 1 , wherein the lines are corrupted by a mask, and a line segment masking (LSM) algorithm is used to infer missed line segments with a flexible wireframe parsing.

4. The system of claim 3 , wherein the flexible wireframe parsing comprises learning an LSM-HAWP network by retraining holistically attracted wireframe parsing (HAWP) with irregular and object segmentation masks, and introducing an indicator function to denote a masking probability of each line segment according to the mask as the post-processing of LSM-HAWP.

5. The system of claim 4 , wherein during post-processing of LSM-HAWP, only in inpainting training process or object removal task, the ground-truth images are known in advance.

6. The system of claim 5 , wherein all wireframes are extracted beforehand and filtered by the post-process of LSM-HAWP, rather than an end-to-end training with LSM-HAWP.

7. The system of claim 4 , wherein line maps are got from connecting junction pairs from LSM-HAWP with anti-aliased lines.

8. The system of claim 1 , wherein edge maps are binary maps.

9. The system of claim 1 , wherein the pyramid structure sub-encoder includes partially gated convolutions blocks, dilated residual blocks, an attention block, and pyramid decomposing separable blocks.

10. The system of claim 9 , wherein in the partially gated convolutions (GC) blocks only three GC layers are used for the input and output features in both upsampling and downsampling parts.

11. The system of claim 9 , wherein the attention block is in the middle of a pyramid structure sub-encoder.

12. The system of claim 9 , wherein the pyramid decomposing separable blocks leverages dilated lower-scale structures.

13. The system of claim 9 , wherein the pyramid decomposing separable blocks project the image feature into two separated embedding spaces; one is a line embedding space and the other is an edge embedding space.

14. The system of claim 1 , wherein the decoder includes partially gated convolutions blocks, dilated residual blocks, without attention blocks or pyramid decomposing separable blocks.

15. A method for image inpainting with an encoder, a decoder, and a sketch tensor space of a third-order tensor, wherein the encoder has a wireframe parser, canny detector, and a pyramid structure sub-encoder, the method comprising:
extracting line maps from an original image input to the encoder with the wireframe parser;
extracting edge maps from the original image with the canny detector;
generating a sketch tensor space with the pyramid structure sub-encoder based on the original image, the line maps and the edge maps; and
outputting with the decoder an inpainted image from the sketch tensor space.

Assignees

• Univ Fudan

Inventors

• Fu Yanwei
• Cao Chenjie

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T5/30

  Erosion or dilatation, e.g. thinning · CPC title