Inpainting via an encoding and decoding network

Patent metadata

Field	Value
Publication number	US-11328392-B2
Application number	US-202017017177-A
Country	US
Kind code	B2
Filing date	Sep 10, 2020
Priority date	Oct 25, 2019
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

An image processing apparatus including: at least one memory; and at least one processor coupled to the at least one memory and configured to implement: an image acquisition module configured to acquire an input image including an object region; a mask image generation module configured to generate a mask image based on the input image; and an image inpainting module configured to extract a fusion feature map corresponding to the input image using an encoding network according to the input image and the mask image, and to inpaint the object region in the input image using a decoding network based on the fusion feature map, to obtain an inpainting result.

First claim

What is claimed is:

1. An image processing apparatus, comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to implement:
an image acquisition module configured to acquire an input image including an object region;
a mask image generation module configured to generate a mask image based on the input image; and
an image inpainting module configured to extract a fusion feature map corresponding to the input image using an encoding network according to the input image and the mask image, and to inpaint the object region in the input image using a decoding network based on the fusion feature map, to obtain an inpainting result,

wherein at least one of the encoding network and the decoding network comprises at least one depth-gated convolution processor.

2. The apparatus according to claim 1 , wherein the encoding network and the decoding network comprise at least one first convolution processor,
wherein the at least one first convolution processor is configured to perform convolution according to an input fusion feature map and outputs the fusion feature map obtained by the convolution.

3. The apparatus according to claim 2 , wherein the encoding network further comprises at least one second convolution processor cascaded with a last first convolution processor of the encoding network,
wherein the at least one second convolution processor is configured to perform dilated convolution according to the input fusion feature map, and output the fusion feature map obtained by the dilated convolution.

4. The apparatus according to claim 3 , wherein the at least one second convolution processor comprises a first-second convolution processor and a second-second convolution processor,
wherein the first-second convolution processor is cascaded in sequence with the second-second convolution processor, and
wherein a first convolution parameter of the first-second convolution processor is different from a second convolution parameter of the second-second convolution processor.

5. The apparatus according to claim 2 , wherein the input fusion feature map is based on a plurality of channels, and
wherein the at least one first convolution processor is further configured to:
perform first convolution according to the input fusion feature map to extract a corresponding image feature map;
perform second convolution based on the input fusion feature map to extract a mask feature map based on at least one channel, wherein a number of the at least one channel is smaller than a number of the plurality of channels;
fuse the image feature map and the mask feature map; and

output a result of the fusing.

6. The apparatus according to claim 5 , wherein before the performing the second convolution, the at least one first convolution processor is further configured to, based on the number of the at least one channel being different from the number of the plurality of channels, convert the input fusion feature map into a converted fusion feature map based on the at least one channel, and
wherein the second convolution is performed based on the converted feature map.

7. The apparatus according to claim 5 , wherein the at least one first convolution processor is further configured to:
perform processing according to at least two convolution processing parameters, and extract feature maps corresponding to at least two receptive fields, based on the input fusion feature map; and
fuse the extracted feature maps corresponding to the at least two receptive fields to obtain the mask feature map.

8. The apparatus according to claim 1 , wherein the image inpainting module is further configured to:
obtain a preliminary inpainting result based on the input image and the mask image;
generate a noise image having a same size as the input image; and
obtain the inpainting result based on the preliminary inpainting result and the noise image.

9. The apparatus according to claim 1 , wherein the image inpainting module is further configured to process an object map by at least one of randomly exchanging element values of element points in adjacent locations in the object map, and randomly adjusting the element values of the element points in the object map,
wherein the object map comprises at least one of the fusion feature map and the inpainting result.

10. The apparatus according to claim 9 , wherein the randomly exchanging comprises:
performing a first edge clipping on the object map to obtain a first clipped map and a second edge clipping on the object map to obtain a second clipped map;
generating a first weight map corresponding to the first clipped map and a second weight map corresponding to the second clipped map, wherein element values of element points in the first weight map and the second weight map are one of 1 or 0, and wherein a first element value of a first element point at a first position of the first weight map is different from a second element value of a second element point at a second position of the second weight map corresponding to the first position; and
fusing the first clipped map and the second clipped map based on the first weight map and the second weight map to obtain a processed map having a same size as the object map.

11. The apparatus according to claim 9 , wherein the randomly adjusting comprises:
performing a third edge clipping on the object map to obtain a third clipped map and a fourth edge clipping on the object map to obtain a fourth clipped map;
performing feature extraction based on the third clipped map to obtain an adjustment coefficient of the fourth clipped map; and
adjusting element values of element points in the fourth clipped map based on the adjustment coefficient to obtain a processed object map with the same size as the object map.

12. The apparatus according to claim 1 , wherein the image acquisition module is further configured to:
acquire an original image including the object region;
extract image features of the original image; and
perform clipping on the original image based on the image features of the original image to obtain the input image including the object region.

13. The apparatus according to claim 12 , wherein the image acquisition module is further configured to:
determine a region size of the object region;
based on the region size being smaller than or equal to a threshold size, obtain candidate regions having a first set region size according to the image features of the original image and location information of the object region;
based on the region size being larger than the threshold size, obtain candidate regions having a second set region size according to the image features of the original image and the location information of the object region;
screen the object region from candidate regions; and
based on the screened object region being a candidate region having the second set region size, clip the original image according to the screened object region to obtain a clipped image having the second set region size, and scaling the clipped image according to the first set region size to obtain the input image, and
wherein the image inpainting module is further configured to:
scale the inpainting result to obtain a scaled inpainting result having the second set region size; and
fuse the scaled inpainting result and the original image to obtain an inpainting result corresponding to the original image.

14. An image processing method, comprising:
acquiring an input image including an object region;
generating a mask image based on the input image;
extracting a fusion feature map corresponding

Assignees

• Samsung Electronics Co Ltd

Inventors

• Bai Mengmeng
• Zuo Li
• Song Xiansong
• Na Jaekeun
• Wang Changwei
• Zhang Weihua

Classifications

• G06V10/806

  of extracted features · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06F18/251

  of input or preprocessed data · CPC title

• G06F18/253

  of extracted features · CPC title