Generating an inpainted image from a masked image using a patch-based encoder

Patent metadata

Field	Value
Publication number	US-12148131-B2
Application number	US-202217733634-A
Country	US
Kind code	B2
Filing date	Apr 29, 2022
Priority date	Apr 29, 2022
Publication date	Nov 19, 2024
Grant date	Nov 19, 2024

Abstract

The disclosure herein describes generating an inpainted image from a masked image using a patch-based encoder and an unquantized transformer. An image including a masked region and an unmasked region is received, and the received image is divided into a plurality of patches including masked patches. The plurality of patches is encoded into a plurality of feature vectors, wherein each patch is encoded to a feature vector. Using a transformer, a predicted token is generated for each masked patch using a feature vector encoded from the masked patch, and a quantized vector of the masked patch is determined using generated predicted token and a masked patch-specific codebook. The determined quantized vector of the masked patch is included into a set of quantized vectors associated with the plurality of patches, and an output image is generated from the set of quantized vectors using a decoder.

First claim

What is claimed is:

1. A system comprising:
at least one processor; and
at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the at least one processor to:
receive an image including a masked region and an unmasked region;
divide the received image into a plurality of patches including a masked patch, wherein the masked patch includes at least a portion of the masked region of the image;
encode the plurality of patches into a plurality of feature vectors, wherein each patch is encoded to a feature vector;
generate a predicted token for the masked patch using a feature vector encoded from the masked patch, wherein the feature vector is unquantized;
determine a quantized vector of the masked patch using at least the generated predicted token;
include the determined quantized vector of the masked patch into a set of quantized vectors associated with the plurality of patches; and
generate an output image from the set of quantized vectors, whereby the output image includes the unmasked region of the received image and image inpainting in a region corresponding to the masked region in the received image.

2. The system of claim 1 , wherein determining the quantized vector of the masked patch further uses a masked patch-specific codebook with the generated predicted token;
wherein the plurality of patches further includes an unmasked patch, wherein the unmasked patch includes no portion of the masked region of the image; and
wherein the at least one memory and the computer program code is configured to, with the at least one processor, further cause the at least one processor to:
determine a token for the unmasked patch using an unmasked patch-specific codebook and a feature vector of the plurality of feature vectors encoded from the unmasked patch;
determine a quantized vector of the unmasked patch using the unmasked patch-specific codebook and the determined token for the unmasked patch;
include the determined quantized vector of the unmasked patch into the set of quantized vectors associated with the plurality of patches; and
wherein generating the output image from the set of quantized vectors further includes inserting the unmasked patch into the output image in a location corresponding to a location of the unmasked patch in the received image.

3. The system of claim 2 , wherein the masked patch-specific codebook includes a map of token values to quantized latent vectors that is generated using machine learning with masked patch data used as training data; and
wherein the unmasked patch-specific codebook includes a map of token values to quantized latent vectors that is generated using machine learning with unmasked patch data used as training data.

4. The system of claim 1 , wherein the plurality of patches includes a plurality of masked patches; and
wherein generating the predicted token for the masked patch using the feature vector encoded from the masked patch further includes:
generating a first plurality of predicted tokens for the plurality of masked patches using feature vectors of the plurality of masked patches in the plurality of feature vectors;
selecting a first token of the generated first plurality of predicted tokens that has a highest probability value of the generated first plurality of predicted tokens, wherein the first token is associated with a first masked patch of the plurality of masked patches;
determining a quantized vector associated with the selected first token;
replacing a feature vector from which the first token was generated with the determined quantized vector associated with the selected first token in the plurality of feature vectors;
generating a second plurality of predicted tokens for the plurality of masked patches using the plurality of feature vectors with the determined quantized vector associated with the selected first token with which the feature vector from which the first token was generated was replaced; and
selecting a second token of the generated second plurality of predicted tokens that has a highest probability value of the generated second plurality of predicted tokens, wherein the second token is associated with a second masked patch of the plurality of masked patches.

5. The system of claim 4 , wherein selecting the first token of the generated first plurality of predicted tokens further includes sampling the first token from the generated first plurality of predicted tokens using Gibbs sampling.

6. The system of claim 1 , wherein the at least one memory and the computer program code is configured to, with the at least one processor, further cause the at least one processor to:
train a decoder to generate the output image from the set of quantized vectors using machine learning, the training including:
processing quantized vectors associated with an input image using a plurality of deconvolutional layers associated with a set of multiple scales;
extracting multi-scale feature maps from a reference image, wherein a feature map of the multi-scale feature maps is extracted for each scale of the set of multiple scales;
fusing the extracted multi-scale feature maps with the processed quantized vectors at each scale of the set of multiple scales;
generating inpainted image data based on the fused multi-scale feature maps and quantized vectors; and
tuning the decoder based on comparison of the generated inpainted image data to corresponding image data of the input image.

7. The system of claim 1 , wherein the at least one memory and the computer program code is configured to, with the at least one processor, further cause the at least one processor to:
train a transformer to generate the predicted token for the masked patch using machine learning, the training including:
generating a set of ground truth feature vectors from an input image, wherein the input image is unmasked;
generating quantized vectors of a subset of the set of ground truth feature vectors using a masked patch-based codebook;
replacing the subset of ground truth feature vectors with the generated quantized vectors in the set of ground truth feature vectors;
processing the set of ground truth feature vectors with the transformer; and
tuning the transformer based on results of processing the set of ground truth feature vectors with the transformer.

8. A computerized method comprising:
receiving, by a processor, an image including a masked region and an unmasked region;
dividing, by the processor, the received image into a plurality of patches including a masked patch, wherein the masked patch includes at least a portion of the masked region of the image;
encoding, by the processor, the plurality of patches into a plurality of feature vectors, wherein each patch is encoded to a feature vector;
generating, by the processor, a predicted token for the masked patch using a feature vector encoded from the masked patch, wherein the feature vector is unquantized;
determining, by the processor, a quantized vector of the masked patch using at least the generated predicted token;
including, by the processor, the determined quantized vector of the masked patch into a set of quantized vectors associated with the plurality of patches; and
generating, by the processor, an output image from the set of quantized vectors, whereby the output image includes image inpainting in a region corresponding to the masked region in the received image.

9. The computerized method of claim 8 , wherein determining the quantized vector of the masked patch further uses a masked patch-specific codebook with the generated predicted token;
wherein the plurality of patches further includes an unmas

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chen Dongdong
• Dai Xiyang
• Chen Yinpeng
• Liu Mengchen
• Yuan Lu

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title