Utilizing deep learning for boundary-aware image segmentation

Patent metadata

Field	Value
Publication number	US-9972092-B2
Application number	US-201615086590-A
Country	US
Kind code	B2
Filing date	Mar 31, 2016
Priority date	Mar 31, 2016
Publication date	May 15, 2018
Grant date	May 15, 2018

Abstract

Systems and methods are disclosed for segmenting a digital image to identify an object portrayed in the digital image from background pixels in the digital image. In particular, in one or more embodiments, the disclosed systems and methods use a first neural network and a second neural network to generate image information used to generate a segmentation mask that corresponds to the object portrayed in the digital image. Specifically, in one or more embodiments, the disclosed systems and methods optimize a fit between a mask boundary of the segmentation mask to edges of the object portrayed in the digital image to accurately segment the object within the digital image.

First claim

We claim:

1. In a digital medium environment for editing digital visual media, a method of using deep learning to segment objects from digital visual media, the method comprising:
generating, by at least one processor with a first neural network comprising a first machine learning model trained using a first plurality of images, a probability map for an input image, wherein the probability map indicates object pixels predicted to correspond to an object portrayed in the input image;
generating, by the at least one processor with a second neural network comprising a second machine learning model trained using a second plurality of images, a boundary map for the input image, wherein the boundary map indicates edge pixels predicted to correspond to edges of the object portrayed in the input image;
based on the probability map and the boundary map, generating, by the at least one processor, a segmentation mask for the object by optimizing a fit between a mask boundary of the object and the edges of the object; and
based on the segmentation mask, identifying, by the at least one processor, a set of pixels corresponding to the object portrayed in the input image.

2. The method of claim 1 , wherein optimizing the fit between the mask boundary of the object and the edges of the object comprises an iterative optimization process using a combination of color modeling and boundary modeling.

3. The method of claim 1 , further comprising generating a boundary refinement map to determine boundary refinement pixels using the probability map and the boundary map, wherein the boundary refinement pixels comprise pixels identified both as object pixels in the probability map and as edge pixels in the boundary map.

4. The method of claim 3 , further comprising:
analyzing areas of the input image corresponding to the boundary refinement pixels; and
wherein optimizing the fit between the mask boundary of the segmentation mask and the edges of the object comprises iteratively fitting the mask boundary to the edges of the object based on analyzing the areas of the input image corresponding to the boundary refinement pixels.

5. The method of claim 4 , wherein analyzing the areas of the input image corresponding to the boundary refinement pixels comprises performing a color modeling function on the areas using Gaussian mixture models.

6. The method of claim 1 , wherein the first plurality of images comprises a first plurality of digital training image pairs, wherein each digital training image pair of the first plurality of digital training image pairs comprises an object probability digital training image and a ground mask that identifies pixels corresponding to an object within the object probability digital training image.

7. The method of claim 6 , wherein the second plurality of images comprises a second plurality of digital training image pairs, wherein each digital training image pair of the second plurality of digital training image pairs comprises an object boundary digital training image and a boundary mask that identifies pixels corresponding to edges of an object within the object boundary digital training image.

8. The method of claim 7 , wherein training the first neural network and training the second neural network comprises fine-tuning the first neural network and fine-tuning the second neural network based on the first plurality of digital training image pairs and the second plurality of digital training image pairs having a related data domain.

9. The method of claim 1 , further comprising:
detecting the object portrayed in the input image;
determining a bounded area within which the object portrayed in the image is located;
creating a cropped portion of a portion of the input image corresponding to the bounded area;
providing the cropped portion to the first neural network to use in generating the probability map; and
providing the cropped portion to the second neural network to use in generating the boundary map.

10. In a digital medium environment for editing digital visual media, a method of using deep learning to segment objects from the digital visual media, the method comprising:
detecting, by at least one processor, an object portrayed within an input image;
identifying a bounded area of the input image comprising a cropped portion of the input image around the detected object;
analyzing the cropped portion of the input image to determine, by the at least one processor, that a portion of the object is located outside of the bounded area based on detecting an indication within a first refinement map;
based on determining the portion of the object is located outside of the bounded area, expanding, by the at least one processor, the bounded area within the input image to include the portion of the object;
analyzing the expanded bounded area within the input image to determine that the entire object is located within the expanded bounded area based on detecting an indication within a second refinement map; and
in response to determining that the entire object is located within the expanded bounded area, optimizing, by the at least one processor, a segmentation mask for the object by optimizing a fit between a mask boundary of the segmentation mask and edges of the object using a cropped portion of the input image corresponding to the expanded bounded area.

11. The method of claim 10 , further comprising:
generating a first probability map using a first neural network;
generating a first boundary map using a second neural network; and
fusing the first probability map with the first boundary map to generate the first refinement map.

12. The method of claim 11 , further comprising generating, using the cropped portion of the input image, a second refinement map, wherein generating the second refinement map comprises:
generating a second probability map using the first neural network;
generating a second boundary map using the second neural network; and
fusing the second probability map with the second boundary map to generate the second refinement map.

13. The method of claim 12 , further comprising generating an initial iteration of the segmentation mask using the second refinement map.

14. The method of claim 13 , wherein optimizing the segmentation mask comprises generating a second iteration of the segmentation mask, wherein the second iteration of the segmentation mask adjusts the fit between the mask boundary and the edges of the object.

15. The method of claim 14 , further comprising:
comparing the second iteration of the segmentation mask with the initial iteration to determine a variance between the second iteration of the segmentation mask and the initial iteration of the segmentation mask; and
based on the variance, determining whether the segmentation mask is optimized.

16. The method of claim 10 , wherein detecting an indication within the first refinement map to determine the portion of the object is outside the bounded area comprises detecting that at least one boundary refinement pixel identified in the first refinement map is positioned on an edge of the first refinement map.

17. The method of claim 16 , wherein expanding the bounded area comprises increasing the size of the bounded area in a direction associated with the edge of the refinement map on which the at least one boundary refinement pixel is positioned.

18. A system for identifying objects within digital visual media, comprising:
at least one processor; and
at least one non-transitory computer readable storage medium storing instructions thereon, that, whe

Assignees

• Adobe Systems Inc

Inventors

• Lin Zhe
• Song Yibing
• Lu Xin
• Shen Xiaohui
• Yang Jimei

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T7/12

  Edge-based segmentation · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T7/13

  Edge detection · CPC title