Deep salient content neural networks for efficient digital object segmentation

Patent metadata

Field	Value
Publication number	US-10460214-B2
Application number	US-201715799395-A
Country	US
Kind code	B2
Filing date	Oct 31, 2017
Priority date	Oct 31, 2017
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

Systems, methods, and non-transitory computer-readable media are disclosed for segmenting objects in digital visual media utilizing one or more salient content neural networks. In particular, in one or more embodiments, the disclosed systems and methods train one or more salient content neural networks to efficiently identify foreground pixels in digital visual media. Moreover, in one or more embodiments, the disclosed systems and methods provide a trained salient content neural network to a mobile device, allowing the mobile device to directly select salient objects in digital visual media utilizing a trained neural network. Furthermore, in one or more embodiments, the disclosed systems and methods train and provide multiple salient content neural networks, such that mobile devices can identify objects in real-time digital visual media feeds (utilizing a first salient content neural network) and identify objects in static digital images (utilizing a second salient content neural network).

First claim

What is claimed is:

1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computer system to:
receive, by a mobile device, a digital image portraying one or more salient objects;
select, by the mobile device, a neural network from a set of neural networks comprising a static salient content neural network and a real-time salient content neural network based on whether the digital image is static or is part of a real-time digital visual media feed;
identify, by the mobile device, the one or more salient objects portrayed in the digital image by applying the selected neural network to the digital image; and
generate a modified digital image based on the identified one or more salient objects portrayed in the digital image.

2. The non-transitory computer readable medium of claim 1 , wherein the digital image is static and further comprising instructions that, when executed by the at least one processor, cause the computer system to:
access the static salient content neural network on the mobile device; and
apply the static salient content neural network to the digital image to identify the one or more salient objects portrayed in the digital image.

3. The non-transitory computer readable medium of claim 1 , wherein the digital image is part of the real-time digital visual media feed and further comprising instructions that, when executed by the at least one processor, cause the computer system to:
access the real-time salient content neural network on the mobile device; and
apply the real-time salient content neural network to the real-time digital visual media feed to identify the one or more salient objects portrayed in the real-time digital visual media feed; and
generate a modified digital visual media feed based on the identified one or more salient objects portrayed in the real-time digital visual media feed.

4. The non-transitory computer readable medium of claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computer system to generate the modified digital visual media feed by:
segregating background pixels of the real-time digital visual media feed from the identified one or more salient objects portrayed in the real-time digital visual media feed; and
modifying at least one of the background pixels of the real-time digital visual media feed or the identified one or more salient objects portrayed in the real-time digital visual media feed.

5. The non-transitory computer readable medium of claim 3 , wherein the real-time salient content neural network has a faster processing speed than the static salient content neural network.

6. The non-transitory computer readable medium of claim 4 , further comprising instructions that, when executed by at least one processor, cause the computer system to:
capture the digital image from the real-time digital visual media feed;
store the digital image on the mobile device; and
identify the one or more salient objects portrayed in the digital image by applying the static salient content neural network to the digital image from the real-time digital visual media feed stored on the mobile device.

7. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to generate the modified digital image by segregating background pixels of the digital image from the identified one or more salient objects portrayed in the digital image.

8. The non-transitory computer readable medium of claim 7 , further comprising instructions that, when executed by the at least one processor, cause the computer system to generate the modified digital image by modifying at least one of the background pixels of the digital image or the identified one or more salient objects.

9. A system for training neural networks to identify objects within digital visual media, comprising:
at least one processor; and
at least one non-transitory computer readable storage medium storing instructions thereon, that, when executed by the at least one processor, cause the system to:
train a real-time salient content neural network by utilizing the real-time salient content neural network to generate predicted foreground pixels of a first training digital image and comparing ground truth foreground pixels of the first training digital image with the predicted foreground pixels of the first training digital image;
train a static salient content neural network by utilizing the static salient content neural network to generate predicted foreground pixels of a second training digital image and comparing ground truth foreground pixels of the second training digital image with the predicted foreground pixels of the second training digital image; and
provide the real-time salient content neural network and the static salient content neural network to a mobile device for identifying one or more salient objects portrayed in a digital image, such that the real-time salient content neural network and the static salient content neural network are selectable on the mobile device based on whether the digital image is static or is part of a real-time digital visual media feed.

10. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to:
train an additional salient content neural network utilizing a third training digital image to determine optimization parameters; and
utilize the optimization parameters of the additional salient content neural network to train the real-time salient content neural network and the static salient content neural network.

11. The system of claim 10 , wherein the optimization parameters comprise at least one of: a weight decay parameter; an initialization parameter; and an iterations parameter.

12. The system of claim 9 , wherein the real-time salient content neural network has a faster processing speed than the static salient content neural network.

13. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to:
access a trained object classification neural network having at least one object classification channel and a background classification channel; and
train the real-time salient content neural network and the static salient content neural network by utilizing the at least one object classification channel and the background classification channel as initialization parameters for the real-time salient content neural network and the static salient content neural network.

14. The system of claim 13 :
wherein the at least one object classification channel comprises a person classification channel and further comprising instructions that, when executed by the at least one processor, cause the system to train the real-time salient content neural network by:
initializing foreground weights of the real-time salient content neural network from the person classification channel of the trained object classification neural network; and
initialize background weights of the real-time salient content neural network from the background classification channel of the trained object classification neural network.

15. In a digital medium environment for generating modified digital visual media, a computer-implemented method for training neural networks to identify objects within digital visual media, comprising:
training a real-time salient content neural network by utilizing the real-time salient content neural network to generate

Assignees

• Adobe Inc

Inventors

• Lu Xin
• Lin Zhe
• Shen Xiaohui
• Yang Jimei
• Zhang Jianming
• Chien Jen-Chan Jeff
• Liu Chenxi

Classifications

• G06V10/17

  using hand-held instruments · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title