Automatically generating a trimap segmentation for a digital image by utilizing a trimap generation neural network

Patent metadata

Field	Value
Publication number	US-11393100-B2
Application number	US-202016988036-A
Country	US
Kind code	B2
Filing date	Aug 7, 2020
Priority date	Aug 7, 2020
Publication date	Jul 19, 2022
Grant date	Jul 19, 2022

Abstract

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing a plurality of neural networks in a multi-branch pipeline to generate image masks for digital images. Specifically, the disclosed system can classify a digital image as a portrait or a non-portrait image. Based on classifying a portrait image, the disclosed system can utilize separate neural networks to generate a first mask portion for a portion of the digital image including a defined boundary region and a second mask portion for a portion of the digital image including a blended boundary region. The disclosed system can generate the mask portion for the blended boundary region by utilizing a trimap generation neural network to automatically generate a trimap segmentation including the blended boundary region. The disclosed system can then merge the first mask portion and the second mask portion to generate an image mask for the digital image.

First claim

What is claimed is:

1. A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to:
generate, utilizing a semantic cropping neural network, a cropped digital image from a portion of a digital image including an object;
extract an estimated foreground region and an estimated background region of the cropped digital image, wherein the estimated foreground region portrays the object in the cropped digital image;
determine, utilizing a trimap generation neural network, a blended boundary region of the object in the cropped digital image, wherein the blended boundary region comprises a combination of foreground elements and background elements;
determine a trimap segmentation of the cropped digital image by combining the estimated foreground region, the estimated background region, and the blended boundary region; and
generate, utilizing the trimap segmentation of the cropped digital image, an image mask for the digital image.

2. The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
resize the cropped digital image to a first resolution size; and
determine, utilizing the trimap generation neural network, the blended boundary region of the object from the cropped digital image at the first resolution size.

3. The non-transitory computer readable storage medium as recited in claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
extract the estimated foreground region and the estimated background region from the cropped digital image at the first resolution size; and
determine the trimap segmentation of the digital image by combining the estimated foreground region, the estimated background region, and the blended boundary region at the first resolution size.

4. The non-transitory computer readable storage medium as recited in claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
upsample the cropped digital image and the trimap segmentation to a second resolution size; and
utilize a matting algorithm to generate the image mask from the upsampled cropped digital image and the upsampled trimap segmentation at the second resolution size.

5. The non-transitory computer readable storage medium as recited in claim 4 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the image mask from the upsampled cropped digital image and the upsampled trimap segmentation at the second resolution size by:
utilizing the matting algorithm to generate an initial image mask at the second resolution size; and
upsampling the initial image mask to an original resolution of the digital image.

6. The non-transitory computer readable storage medium as recited in claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the image mask by utilizing a guided image filter to refine a boundary between an updated foreground region and an updated background region in the initial image mask.

7. The non-transitory computer readable storage medium as recited in claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the image mask by:
generating an updated foreground region and an updated background region by assigning, utilizing the matting algorithm, alpha values to portions of the blended boundary region based on the trimap segmentation; and
generating the image mask based on the updated foreground region and the updated background region according to the assigned alpha values.

8. The non-transitory computer readable storage medium as recited in claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computing device to combine the image mask with an additional image mask corresponding to an additional portion of the digital image based on an original position of the cropped digital image relative to the digital image.

9. A system comprising:
at least one computer memory device comprising a digital image; and
one or more servers configured to cause the system to:
generate, utilizing a semantic cropping neural network, a cropped digital image from a portion of the digital image including an object;
determine an estimated foreground region and an estimated background region of the cropped digital image, wherein the estimated foreground region portrays the object in the digital image;
determine, utilizing a trimap generation neural network, a blended boundary region of the object in the cropped digital image, wherein the blended boundary region comprises a combination of foreground elements and background elements;
determine a trimap segmentation of the cropped digital image by combining the estimated foreground region, the estimated background region, and the blended boundary region;
generate an updated foreground region and an updated background region by utilizing a matting algorithm to assign an alpha value for each pixel within the blended boundary region of the cropped digital image based on the trimap segmentation; and
generate an image mask for the digital image based on the updated foreground region and the updated background region according to the assigned alpha values.

10. The system as recited in claim 9 , wherein the one or more servers are further configured to cause the system to determine the estimated foreground region and the estimated background region of the digital image by:
resizing the cropped digital image to a first resolution size; and
determining the estimated foreground region and the estimated background region from the cropped digital image at the first resolution size.

11. The system as recited in claim 10 , wherein the one or more servers are further configured to cause the system to determine the blended boundary region by determining, utilizing the trimap generation neural network, the blended boundary region of the object from the cropped digital image at the first resolution size.

12. The system as recited in claim 10 , wherein the one or more servers are further configured to cause the system to:
upsample the cropped digital image and the trimap segmentation to a second resolution size; and
generate the updated foreground region and the updated background region by utilizing the matting algorithm to assign an alpha value to each pixel within the blended boundary region based on the upsampled cropped digital image and the upsampled trimap segmentation at the second resolution size.

13. The system as recited in claim 10 , wherein the one or more servers are further configured to:
determine a position of the cropped digital image relative to the digital image; and
combine the image mask with an additional image mask corresponding to an additional portion of the digital image based on the position of the cropped digital image relative to the digital image.

14. The system as recited in claim 9 , wherein the one or more servers are further configured to cause the system to generate the image mask by:
upsampling an initial image mask generated by the matting algorithm to an original resolution of the digital image; and
refining, utilizing a guided image filter, a boundary between the updated foreground region and the updated background region from the initial image mask at the original reso

Assignees

• Adobe Inc

Inventors

• Zhang He
• Safdarnejad Seyed Morteza
• Wang Yilin
• Wei Zijun
• Zhang Jianming
• Tambe Salil
• Price Brian

Classifications

• G06T7/194 Primary

  involving foreground-background segmentation · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title