Boundary-aware object removal and content fill

Patent metadata

Field	Value
Publication number	US-10825148-B2
Application number	US-201816204226-A
Country	US
Kind code	B2
Filing date	Nov 29, 2018
Priority date	Nov 29, 2018
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

Systems and methods for removing objects from images are disclosed. An image processing application identifies a boundary of each object of a set of objects in an image. In some cases, the identification uses deep learning. The image processing application identifies a completed boundary for each object of the set of objects by providing the object to a trained model. The image processing application determines a set of masks. Each mask corresponds to an object of the set of objects and represents a region of the image defined by an intersection of the boundary of the object and the boundary of a target object to be removed from the image. The image processing application updates each mask by separately performing content filling on the corresponding region. The image processing application creates an output image by merging each of the updated masks with portions of the image.

First claim

The invention claimed is:

1. A method of removing an object from an image, the method comprising:
accessing an image comprising a plurality of objects;
identifying a completed boundary for each object of the plurality of objects by providing each object to a trained adversarial network model;
receiving a selection of a target object from the plurality of objects, the target object to be removed from the image;
determining a set of masks, wherein each mask corresponds to a portion of an object of the plurality of objects and is defined by an intersection of the completed boundary of the corresponding object and the completed boundary of the target object;
updating each mask in the set of masks by separately performing content filling on the corresponding portion of the corresponding object; and
creating an output image by merging each of the updated masks with the corresponding object.

2. The method of claim 1 , wherein portions of the image are regions of the image that are not represented by any of the masks in the set of masks.

3. The method of claim 1 , wherein the trained adversarial network model is a conditional adversarial network.

4. The method of claim 1 , wherein accessing the image further comprises detecting, in the image, the plurality of objects.

5. The method of claim 1 , wherein the content filling comprises applying content-aware fill separately to each mask based on the corresponding object.

6. The method of claim 1 , wherein creating the output image comprises creating a set of layers from the set of masks and merging each layer of the set of layers into the output image.

7. The method of claim 1 , wherein performing the content filling comprises:
dividing each mask into a plurality of segments;
performing, on each segment, content aware fill; and
combining each segment into the respective mask.

8. The method of claim 1 , wherein performing the content filling
comprises:
generating a candidate patch that comprises pixels from the object that is represented by the mask;
validating the candidate patch; and
automatically reconstructing pixels of the mask using the validated candidate patch.

9. A system comprising:
one or more processing devices; and
a non-transitory computer-readable medium communicatively coupled to the one or more processing devices, wherein the one or more processing devices are configured to execute instructions and thereby perform operations comprising:
accessing an image comprising a plurality of objects;
identifying a completed boundary for each object of the plurality of objects by providing each object to a trained adversarial network model;
receiving a selection of a target object from the plurality of objects, the target object to be removed from the image;
determining a set of masks, wherein each mask corresponds to a portion of an object of the plurality of objects and is defined by an intersection of the completed boundary of the corresponding object and the completed boundary of the target object;
updating each mask in the set of masks by separately performing content filling on the corresponding portion of the corresponding object; and
creating an output image by merging each of the updated masks with the corresponding object.

10. The system of claim 9 , wherein portions of the image are regions of the image that are not represented by any of the masks in the set of masks.

11. The system of claim 9 , wherein the trained adversarial network model is a conditional adversarial network.

12. The system of claim 9 , wherein accessing the image further comprises detecting, in the image, the plurality of objects.

13. The system of claim 9 , wherein performing the content filling comprises applying content-aware fill separately to each mask based on the corresponding object.

14. The system of claim 9 , wherein creating the output image comprises creating a set of layers from each of the masks and merging each layer of the set of layers into the output image.

15. The system of claim 9 , wherein performing the content filling comprises:
dividing each mask into a plurality of segments;
performing, on each segment, content aware fill; and
combining each segment into the respective mask.

16. The system of claim 9 , wherein performing the content filling comprises:
generating a candidate patch that comprises pixels from the object that is represented by the mask;
validating the candidate patch; and
automatically reconstructing pixels of the mask using the validated candidate patch.

Assignees

• Adobe Inc

Inventors

• Tagra Sanjeev
• Jain Ajay
• Soni Sachin
• Bedi Ajay

Classifications

• G06F18/00

  Pattern recognition · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T11/40

  Filling planar surfaces by adding surface attributes, e.g. adding colours or textures · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title