Iteratively refining segmentation masks

Patent metadata

Field	Value
Publication number	US-11676283-B2
Application number	US-202217660361-A
Country	US
Kind code	B2
Filing date	Apr 22, 2022
Priority date	Aug 7, 2020
Publication date	Jun 13, 2023
Grant date	Jun 13, 2023

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate refined segmentation masks for digital visual media items. For example, in one or more embodiments, the disclosed systems utilize a segmentation refinement neural network to generate an initial segmentation mask for a digital visual media item. The disclosed systems further utilize the segmentation refinement neural network to generate one or more refined segmentation masks based on uncertainly classified pixels identified from the initial segmentation mask. To illustrate, in some implementations, the disclosed systems utilize the segmentation refinement neural network to redetermine whether a set of uncertain pixels corresponds to one or more objects depicted in the digital visual media item based on low-level (e.g., local) feature values extracted from feature maps generated for the digital visual media item.

First claim

What is claimed is:

1. A method comprising:
receiving a digital visual media item portraying an object;
generating, utilizing a segmentation refinement neural network, an initial segmentation mask for the digital visual media item;
determining, utilizing the segmentation refinement neural network, a set of pixels having an associated classification from the initial segmentation mask; and
generating a refined segmentation mask for the digital visual media item by utilizing the segmentation refinement neural network to iteratively refine the initial segmentation mask by iteratively refining the set of pixels having the associated classification from the initial segmentation mask.

2. The method of claim 1 , wherein:
determining, utilizing the segmentation refinement neural network, the set of pixels having the associated classification from the initial segmentation mask comprises determining, utilizing the segmentation refinement neural network, uncertain pixels having an uncertain classification from the initial segmentation mask; and
utilizing the segmentation refinement neural network to iteratively refine the initial segmentation mask comprises utilizing the segmentation refinement neural network to iteratively refine the uncertain pixels from the initial segmentation mask.

3. The method of claim 2 , wherein determining, utilizing the segmentation refinement neural network, the uncertain pixels from the initial segmentation mask comprises:
generating, utilizing the segmentation refinement neural network, an uncertainty map comprising uncertainty scores for pixels of the initial segmentation mask; and
determining the uncertain pixels from the initial segmentation mask based on the uncertainty scores of the uncertainty map.

4. The method of claim 1 , wherein generating the refined segmentation mask for the digital visual media item by utilizing the segmentation refinement neural network to iteratively refine the initial segmentation mask comprises:
generating a first refined segmentation mask for the digital visual media item by utilizing the segmentation refinement neural network to refine the initial segmentation mask; and
generating a second refined segmentation mask for the digital visual media item by utilizing the segmentation refinement neural network to refine the first refined segmentation mask.

5. The method of claim 1 ,
further comprising generating, utilizing the segmentation refinement neural network, a set of feature maps corresponding to the digital visual media item,
wherein generating, utilizing the segmentation refinement neural network, the initial segmentation mask for the digital visual media item comprises generating, utilizing the segmentation refinement neural network, the initial segmentation mask based on the set of feature maps.

6. The method of claim 5 , wherein generating, utilizing the segmentation refinement neural network, the set of feature maps corresponding to the digital visual media item comprises utilizing the segmentation refinement neural network to:
generate an initial feature map from the digital visual media item;
generate an additional initial feature map from the initial feature map; and
generate a final feature map from the additional initial feature map.

7. The method of claim 6 , wherein:
generating the initial feature map comprises generating a first set of low-level feature values corresponding to local attributes of the digital visual media item;
generating the additional initial feature map comprises generating a second set of low-level feature values corresponding to the local attributes of the digital visual media item, the second set of low-level feature values comprising a higher level of feature values than the first set of low-level feature values; and
generating the final feature map comprises generating a set of high-level feature values corresponding to global attributes of the digital visual media item.

8. The method of claim 1 , wherein:
generating, utilizing the segmentation refinement neural network, the initial segmentation mask for the digital visual media item comprises generating, utilizing the segmentation refinement neural network the initial segmentation mask indicating a location and a rough shape of the object portrayed in the digital visual media item; and
generating the refined segmentation mask for the digital visual media item by utilizing the segmentation refinement neural network to iteratively refine the initial segmentation mask comprises generating the refined segmentation mask by utilizing the segmentation refinement neural network to iteratively recapture local details associated with the object portrayed in the digital visual media item.

9. The method of claim 1 , further comprising:
providing the digital visual media item and one or more modification elements for display via a graphical user interface of a computing device;
receiving, via the graphical user interface, a user interaction with the one or more modification elements, wherein generating the initial segmentation mask and the refined segmentation mask is in response to receiving the user interaction;
modifying the digital visual media item utilizing the refined segmentation mask; and
providing the modified digital visual media item for display via the graphical user interface.

10. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving a digital visual media item comprising a plurality of pixels, the digital visual media item portraying an object;
generating, utilizing a segmentation refinement neural network, an initial segmentation mask that classifies pixels from the plurality of pixels as corresponding to the object or not corresponding to the object;
determining uncertain pixels that have an associated uncertainty that the uncertain pixels have been classified correctly within the initial segmentation mask; and
generating a refined segmentation mask for the digital visual media item by utilizing the segmentation refinement neural network to iteratively refine the uncertain pixels.

11. The non-transitory computer-readable medium of claim 10 , wherein utilizing the segmentation refinement neural network to iteratively refine the uncertain pixels comprises utilizing the segmentation refinement neural network to iteratively reclassify at least a subset of uncertain pixels from the uncertain pixels as corresponding to the object or not corresponding to the object.

12. The non-transitory computer-readable medium of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
generating, utilizing the segmentation refinement neural network, an uncertainty map comprising uncertainty scores for pixels of the initial segmentation mask; and
determining a ranking for the uncertainty scores of the uncertainty map,
wherein determining the uncertain pixels comprises determining the uncertain pixels based on the ranking of the uncertainty scores from the uncertainty map.

13. The non-transitory computer-readable medium of claim 12 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
generating, utilizing the segmentation refinement neural network, a feature map from the digital visual media item; and
combining the feature map and the initial segmentation mask to generate a combined map, wherein generating, utilizing the segmentation refinement neural

Assignees

• Adobe Inc

Inventors

• Liu Zichuan
• Zhao Wentian
• Wang Shitong
• Qin He
• Jia Yumin
• Kim Yeojin
• Lu Xin
• Chien Jen-Chan

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title