Missing region prediction

Patent metadata

Field	Value
Publication number	US-9760978-B1
Application number	US-201615149415-A
Country	US
Kind code	B1
Filing date	May 9, 2016
Priority date	May 9, 2016
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

Missing region prediction techniques are described. In implementations, an image pair is obtained that includes first and second images. The first image is corrupted by removing a region of content, resulting in a corrupted image having a missing region. The corrupted image and the second image of the image pair are then used to generate a training-image pair. Then, based on a plurality of training-image pairs including the generated training-image pair, a model is trained using machine learning. The model can subsequently be used to predict pixel values of pixels within a subsequent missing region of a subsequent image that is not used as part of the training.

First claim

What is claimed is:

1. In a digital medium environment to predict pixel values of a missing region in an image, a method implemented by at least one computing device, the method comprising:
obtaining, by the at least one computing device, an image pair including first and second images;
corrupting, by the at least one computing device, the first image by removing a region of content to produce a corrupted first image having a missing region;
generating, by the at least one computing device, a training-image pair including the corrupted first image and the second image of the image pair; and
training, by the at least one computing device, a model using machine learning on a plurality of training-image pairs including the training-image pair, the model trained to predict pixel values of pixels within a subsequent missing region of a subsequent image that is not used as part of the training.

2. A method as recited in claim 1 , wherein the missing region is surrounded by non-uniform content.

3. A method as recited in claim 1 , wherein the region is removed by setting pixel values of pixels within the missing region to zero.

4. A method as recited in claim 1 , wherein the machine learning uses a neural network.

5. A method as recited in claim 1 , wherein the machine learning includes end-to-end and pixel-to-pixel learning on the plurality of training-image pairs to train the model without relying on nearest neighbor information.

6. A method as recited in claim 1 , wherein the training is performed by using the corrupted first image as an input and the second image as an output.

7. A method as recited in claim 1 , wherein the training includes configuring the model to encode image semantics and structural information associated with at least one of the corrupted first image or the second image.

8. A method as recited in claim 1 , further comprising controlling a content aware fill operation using the predicted pixel values as initialization values.

9. In a digital medium environment to predict pixel values of a missing region in an image, a method implemented by at least one processor, the method comprising:
obtaining a plurality of image pairs, each image pair including an original image and a corrupted image that is a corrupted version of the original image, the corrupted image having at least one region of missing content;
training, by the at least one processor, a model using pixel-wise end-to-end machine learning on the plurality of images pairs based on the corrupted image as input and the original image as output, the model trained to identify content corresponding to the missing content and restore the corrupted image to the original image; and
generating, based on the model, initial predicted pixel values corresponding to pixels within a subsequent missing region of a subsequent corrupted image that is independent of the plurality of image pairs.

10. A method as recited in claim 9 , further comprising corrupting the corrupted image in respective image pairs of the plurality of image pairs by removing content from the at least one region.

11. A method as recited in claim 10 , wherein the at least one region is removed by setting pixel values of pixels within the region to zero.

12. A method as recited in claim 9 , further comprising using the initial predicted pixel values as initialization values for a content aware fill operation.

13. A method as recited in claim 9 , further comprising configuring the model to encode image semantics and structural information associated with at least one of the corrupted image or the original image.

14. A method as recited in claim 9 , wherein the model comprises a fully convolutional neural network.

15. In a digital medium environment to predict content of a missing region in an image to enhance results of a content aware fill operation, a system comprising:
at least one processor; and
at least one computer-readable storage media storing instructions that are executable by the at least one processor to implement a similarity module configured to:
obtain an image having an empty region lacking content;
apply a neural network to the image to predict content for the empty region, the neural network trained using machine learning on a plurality of image pairs independent of the image, each said image pair having an original image and a corrupted image that is a corrupted version of the original image;
identify at least one additional region, in the image, that includes content having visual similarities to the predicted content in the empty region; and
generate prediction results that map the content from the at least one additional region to the empty region.

16. A system as recited in claim 15 , wherein the content from the at least one additional region is used to patch the empty region.

17. A system as recited in claim 15 , wherein:
the similarity module is further configured to generate a distance map for each said image pair to train the neural network; and
the distance map describes visual similarities between a missing region in the corrupted image and remaining regions in the corrupted image.

18. A system as recited in claim 17 , wherein the distance map includes pixels that each represent a similarity between an area centered at the pixel and the missing region in the corrupted image.

19. A system as recited in claim 15 , wherein the region and the at least one additional region are a same size.

20. A system as recited in claim 15 , wherein the model comprises a fully convolutional neural network.

Assignees

• Adobe Systems Inc

Inventors

• Lu Xin
• Lin Zhe
• Chien Jen-Chan Jeff

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T5/006

  Physics · mapped topic