Iteratively applying neural networks to automatically identify pixels of salient objects portrayed in digital images

Patent metadata

Field	Value
Publication number	US-11244195-B2
Application number	US-201815967928-A
Country	US
Kind code	B2
Filing date	May 1, 2018
Priority date	May 1, 2018
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

The present disclosure relates to systems, method, and computer readable media that iteratively apply a neural network to a digital image at a reduced resolution to automatically identify pixels of salient objects portrayed within the digital image. For example, the disclosed systems can generate a reduced-resolution digital image from an input digital image and apply a neural network to identify a region corresponding to a salient object. The disclosed systems can then iteratively apply the neural network to additional reduced-resolution digital images (based on the identified region) to generate one or more reduced-resolution segmentation maps that roughly indicate pixels of the salient object. In addition, the systems described herein can perform post-processing based on the reduced-resolution segmentation map(s) and the input digital image to accurately determine pixels that correspond to the salient object.

First claim

What is claimed is:

1. A system comprising:
at least one processor; and
a non-transitory computer readable medium storing instructions thereon, that, when executed by at least one processor, cause the system to:
generate a reduced-resolution digital image from an input digital image having an image resolution;
apply a neural network to the reduced-resolution digital image to identify a region of the input digital image corresponding to a salient object portrayed within the input digital image;
based on the identified region and the input digital image, generate a digital image of the region by processing the input digital image to isolate the identified region portraying the salient object;
generate a reduced-resolution digital image of the region from the digital image of the region;
apply the neural network to the reduced-resolution digital image of the region portraying the salient object to generate a reduced-resolution segmentation map of the salient object; and
based on the reduced-resolution segmentation map and the input digital image, generate a segmentation map for selecting the salient object in the digital image, wherein the segmentation map has a resolution corresponding to the image resolution of the input digital image.

2. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to apply the neural network to the reduced-resolution digital image to identify a region of the input digital image corresponding to the salient object by:
utilizing the neural network to generate an initial segmentation map for the reduced-resolution digital image, wherein the initial segmentation map comprises a mapping of pixels within the reduced-resolution digital image that correspond to the salient object; and
identifying the region of the input digital image corresponding to the salient object based on the mapping of pixels from the initial segmentation map.

3. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to:
analyze the reduced-resolution segmentation map of the salient object to identify a sub-region of the input digital image on which additional refinement is needed within the identified region of the input digital image, wherein the sub-region corresponds to a portion of the salient object portrayed within the input digital image;
based on the identified sub-region of the input digital image, generate a reduced-resolution digital image of the sub-region; and
apply the neural network to the reduced-resolution digital image of the sub-region to generate an additional reduced-resolution segmentation map of the portion of the salient object corresponding to the sub-region of the input digital image.

4. The system of claim 3 , wherein the reduced-resolution segmentation map of the salient object comprises confidence values corresponding to pixels within the region of the input digital image and further comprising instructions that, when executed by the at least one processor, cause the system to identify the sub-region based on the confidence values.

5. The system of claim 4 further comprising instructions that, when executed by the at least one processor, cause the system to identify the sub-region by determining that one or more confidence values corresponding to one or more pixels in the sub-region do not satisfy a threshold confidence value.

6. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to:
identify an additional region of the input digital image corresponding to an additional salient object portrayed within the input digital image;
apply the neural network to an additional reduced-resolution digital image of the additional region to generate an additional reduced-resolution segmentation map of the additional salient object; and
generate the segmentation map based on the additional reduced-resolution segmentation map.

7. The system of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to generate the segmentation map of the salient object having the resolution corresponding to the image resolution of the input digital image by applying at least one of a dense conditional random field (CRF) filter, a guided filter, or a graph cut filter to the reduced-resolution segmentation map of the salient object in view of the input digital image to identify pixels of the input digital image corresponding to the salient object.

8. The system of claim 1 , wherein a resolution of the reduced-resolution digital image corresponds to a resolution of the reduced-resolution segmentation map of the salient object.

9. The system of claim 1 , wherein a resolution of the reduced-resolution digital image corresponds to a resolution of the reduced-resolution digital image of the region.

10. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:
apply a neural network to a reduced-resolution digital image of an input digital image to identify a region of the input digital image portraying a salient object;
generate a digital image of the region by processing the input digital image to isolate the identified region portraying the salient object;
generate a second reduced-resolution digital image of the region from the digital image of the region;
generate a reduced-resolution segmentation map of the salient object by applying the neural network to the second reduced-resolution digital image of the region;
based on the reduced-resolution segmentation map and the input digital image, generate a segmentation map of the salient object having a map resolution corresponding to an image resolution of the input digital image; and
select pixels of the input digital image portraying the salient object based on the segmentation map.

11. The non-transitory computer readable medium of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
generate the reduced-resolution digital image of the input digital image by down-sampling the input digital image to a resolution; and
generate the second reduced-resolution digital image of the region of the input digital image by down-sampling the digital image of the region to a resolution corresponding to the resolution of the reduced-resolution digital image of the input digital image.

12. The non-transitory computer readable medium of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the computer system to apply the neural network to the reduced-resolution digital image of the input digital image to identify the region of the input digital image by:
utilizing the neural network to generate an initial segmentation map for the reduced-resolution digital image, wherein the initial segmentation map comprises a mapping of pixels within the reduced-resolution digital image that correspond to the salient object; and
identifying the region of the input digital image corresponding to the salient object based on pixels from the initial segmentation map.

13. The non-transitory computer readable medium of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the computer system to generate the reduced-resolution segmentation map of the salient object by iteratively applying the neural network to one or more additional reduced-resolution digital images within the region until satisfying a threshold condition.

Assignees

• Adobe Inc

Inventors

• Pao I-Ming
• Lin Zhe
• Stuckey Sarah
• Zhang Jianming
• Leong Betty

Classifications

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06V10/7747

  Organisation of the process, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title