Image resynthesis using forward warping, gap discriminators, and coordinate-based inpainting

Patent metadata

Field	Value
Publication number	US-2021334935-A1
Application number	US-201917282214-A
Country	US
Kind code	A1
Filing date	Nov 7, 2019
Priority date	Nov 9, 2018
Publication date	Oct 28, 2021
Grant date	—

Abstract

The invention relates to image processing and, in particular, to image resynthesis for synthesizing new views of a person or an object based on an input image, to resolve tasks such as predicting views of a person or an object from new viewpoints and in new poses. Technical result consists in improved accuracy of image resynthesis based on at least one input image. An image resynthesis system, a system for training a gap filling module to be used in the image resynthesis system, an image resynthesis method, a computer program product and a computer-readable medium are provided. The image resynthesis system comprises a source image input module, a forward warping module configured to predict, for each source image pixel, a corresponding position in a target image, the forward warping module being configured to predict a forward warping field which is aligned with the source image, and a gap filling module configured to fill in the gaps resulting from the application of the forward warping module. The image resynthesis method comprises the steps of: inputting a source image, predicting, for each source image pixel, a corresponding position in a target image, wherein a forward warping field which is aligned with the source image is predicted, predicting a binary mask of gaps which result from the forward warping, filling in the gaps based on said binary mask of gaps by generating a texture image by means of predicting a pair of coordinates in the source image for each pixel in the texture image, and mapping the whole texture back to a new pose using backward warping.

First claim

1 . An image resynthesis system comprising:
a source image input module; a forward warping module configured to predict, for each source image pixel, a corresponding position in a target image, the forward warping module being configured to predict a forward warping field which is aligned with the source image; and a gap filling module configured to fill in the gaps resulting from the application of the forward warping module.

2 . The image resynthesis system according to claim 1 , wherein the gap filling module further comprises a warping error correction module configured to correct forward warping errors in the target image.

3 . The image resynthesis system according to claim 1 , further comprising a texture transfer architecture configured to:
predict warping fields for the source image and the target image; map the source image into a texture space via forward warping; restore the texture space into a whole texture; and map the whole texture back to a new pose using backward warping.

4 . The image resynthesis system according to claim 1 , further comprising a texture extraction module configured to extract texture from the source image.

5 . The image resynthesis system according to claim 1 , wherein at least the forward warping module and the gap filling module are implemented as deep convolutional neural networks.

6 . The image resynthesis system according to claim 1 , wherein the gap filling module comprises a gap inpainter, the gap inpainter comprising:
a coordinate assignment module configured to assign, to each pixel p=(x,y) of the input image, a pair of texture coordinates (u,v) according to a fixed pre-defined texture mapping, so as to provide a two-channel map of x and y values in the texture coordinate frame; a texture map completion module configured to provide a complete texture map, where for each texture pixel (u,v) a corresponding image pixel (x[u,v],y[u,v]) is known; a final texture generating module configured to generate a final texture by mapping image values from positions (x[u,v],y[u,v]) onto the texture at positions (u,v) so as to provide a complete color final texture; and a final texture remapping module configured to remap the final texture to a new view by providing a different mapping from the image pixel coordinates to the texture coordinates.

7 . The image resynthesis system according to claim 5 , wherein at least one of the deep convolutional networks is trained using a real/fake discriminator configured to discriminate ground truth images and inpainted images.

8 . The image resynthesis system according to claim 4 , further comprising an image refinement module configured to correct output image defects.

9 . A system for training a gap filling module configured to fill in gaps as part of image resynthesis, the system being configured to train the gap filling module in parallel and jointly with a gap discriminator network, whereas the gap discriminator network is trained to predict a binary mask of gaps, and the gap filling module is trained to minimize the accuracy the gap discriminator network.

10 . An image resynthesis method comprising the steps of:
inputting a source image; predicting, for each source image pixel, a corresponding position in a target image, wherein a forward warping field which is aligned with the source image is predicted; predicting a binary mask of gaps which result from the forward warping; filling in the gaps based on said binary mask of gaps by generating a texture image by means of predicting a pair of coordinates in the source image for each pixel in the texture image; and mapping the whole texture back to a new pose using backward warping.

11 . The image resynthesis method according to claim 10 , wherein the filling in the gaps comprises the steps of:
assigning, to each pixel p=(x,y) of the input image, a pair of texture coordinates (u,v) according to a fixed pre-defined texture mapping, so as to provide a two-channel map of x and y values in the texture coordinate frame; providing a complete texture map, where for each texture pixel (u,v) a corresponding image pixel (x[u,v],y[u,v]) is known; generating a final texture by mapping image values from positions (x[u,v],y[u,v]) onto the texture at positions (u,v) so as to provide a complete color final texture; and remapping the final texture to a new view by providing a different mapping from the image pixel coordinates to the texture coordinates.

12 . A method for training a gap filling module configured to fill in gaps as part of image resynthesis, the method comprising training the gap filling module in parallel and jointly with a gap discriminator network, whereas the gap discriminator network is trained to predict a binary mask of gaps, and the gap filling module is trained to minimize the accuracy the gap discriminator network.

13 . A computer program product comprising computer program code which, when executed by one or more processors, causes the one or more processors to implement the method according to claim 10 .

14 . A non-transitory computer-readable medium having stored thereon the computer program product according to claim 13 .

15 . A computer program product comprising computer program code which, when executed by one or more processors, causes the one or more processors to implement the method according to claim 11 .

Assignees

• Samsung Electronics Co Ltd

Inventors

• Grigoriev Artur Andreevich
• Lempitsky Victor Sergeevich
• Sevastopolsky Artem Mikhailovich
• VAKHITOV Alexander Timurovich

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title