Utilizing deep learning for automatic digital image segmentation and stylization

Patent metadata

Field	Value
Publication number	US-9978003-B2
Application number	US-201715679989-A
Country	US
Kind code	B2
Filing date	Aug 17, 2017
Priority date	Jan 25, 2016
Publication date	May 22, 2018
Grant date	May 22, 2018

Abstract

Systems and methods are disclosed for segregating target individuals represented in a probe digital image from background pixels in the probe digital image. In particular, in one or more embodiments, the disclosed systems and methods train a neural network based on two or more of training position channels, training shape input channels, training color channels, or training object data. Moreover, in one or more embodiments, the disclosed systems and methods utilize the trained neural network to select a target individual in a probe digital image. Specifically, in one or more embodiments, the disclosed systems and methods generate position channels, training shape input channels, and color channels corresponding the probe digital image, and utilize the generated channels in conjunction with the trained neural network to select the target individual.

First claim

We claim:

1. In a digital medium environment, a method of training at least one neural network to automatically select individuals portrayed in digital visual media, the method comprising:
identifying a training target individual portrayed in a digital training image;
generating a training shape input channel for the digital training image that comprises an estimated shape of the training target individual portrayed in the digital training image;
generating a training color channel reflecting colors of pixels in the digital training image; and
utilizing the training shape input channel and the training color channel to train a neural network to predict pixels portrayed in probe digital images that reflect target individuals.

2. The method of claim 1 , wherein utilizing the training shape input channel and the training color channel to train the neural network comprises:
generating, utilizing the neural network and based on the training shape input channel and the training color channel, an output mask corresponding to the training target individual; and
comparing the output mask to a training ground truth mask corresponding to the training target individual; and
training the neural network based on the comparison between the output mask and the training ground truth mask.

3. The method of claim 2 , wherein:
the neural network comprises a fully convolutional neural network; and
comparing the output mask to the training ground truth mask comprises applying a loss function that measurers error between the output mask and the ground truth mask.

4. The method of claim 1 , wherein generating the training shape input channel comprises:
generating a mean digital object mask based on target individuals in a plurality of digital images, the mean digital object mask comprising a shape corresponding to the target individuals in the plurality of digital images; and
utilizing the mean digital object mask to generate the shape input channel.

5. The method of claim 4 , wherein generating the mean digital object mask comprises:
identifying a set of pixels representing a target individual in a digital image from the plurality of digital images;
identifying one or more facial feature points corresponding to the target individual in the digital image;
estimating a first transform between the facial feature points and a canonical pose; and
applying the first transform to the set of pixels representing the target individual in the digital image.

6. The method of claim 4 , wherein utilizing the mean digital object mask to generate the training shape input channel comprises:
detecting one or more facial feature points corresponding to the training target individual portrayed in the digital training image;
estimating a second transform based on the detected one or more facial feature points corresponding to the training target individual portrayed in the digital training image and the canonical pose; and
applying the second transform to the mean digital object mask to generate the shape input channel.

7. The method of claim 1 , further comprising:
generating a training position channel for the digital training image that indicates positions of pixels in the digital training image relative to the identified training target individual portrayed in the digital training image; and
utilizing the training position channel to train the neural network.

8. The method of claim 7 , wherein generating the training position channel further comprises:
detecting one or more facial feature points corresponding to a face of the training target individual portrayed in the digital training image;
estimating a transform between the detected one or more facial feature points and a canonical pose; and
applying the transform to the canonical pose to generate the training position channel.

9. The method of claim 7 , wherein generating the training position channel further comprises:
generating an x-position channel that indicates horizontal positions of pixels in the digital training image relative to the training target individual portrayed in the digital training image; and
generating a y-position channel that indicates vertical positions of pixels in the digital training image relative to the training target individual portrayed in the digital training image.

10. A system for training at least one neural network to automatically select individuals portrayed in digital visual media, comprising:
at least one processor; and
at least one non-transitory computer readable storage medium storing instructions thereon, that, when executed by the at least one processor, cause the system to:
generate a training position channel for a digital training image portraying a training target individual, the training positing channel indicating positions of pixels in the digital training image relative to the training target individual portrayed in the digital training image;
generate a training color channel reflecting colors of pixels in the digital training image; and
utilize the training position channel and the training color channel to train a neural network to predict pixels portrayed in probe digital images that reflect target individuals.

11. The system of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the system to train the neural network by:
generating, utilizing the neural network and based on the training position channel and the training color channel, an output mask corresponding to the training target individual; and
comparing the output mask to a training ground truth mask corresponding to the training target individual; and
training the neural network based on the comparison between the output mask and the training ground truth mask.

12. The system of claim 11 , wherein:
the neural network comprises a fully convolutional neural network; and
comparing the output mask to the training ground truth mask comprises applying a loss function that measurers error between the output mask and the ground truth mask.

13. The system of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the system to generate the training position channel by:
detecting one or more facial feature points corresponding to a face of the training target individual portrayed in the digital training image;
estimating a transform between the detected one or more facial feature points and a canonical pose; and
applying the transform to the canonical pose to generate the training position channel.

14. The system of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the system to generate the training position channel by:
generating an x-position channel that indicates horizontal positions of pixels in the digital training image relative to the training target individual portrayed in the digital training image; and
generating a y-position channel that indicates vertical positions of pixels in the digital training image relative to the training target individual portrayed in the digital training image.

15. The system of claim 10 , further comprising instructions that, when executed by the at least one processor, cause the system to generate a training shape input channel for each digital training image that comprises an estimated shape of the identified training target individual portrayed in the digital training image.

16. The system of claim 15 , wherein generating the training shape input channel comprises:
detecting one or more facial feature points corresponding to the

Assignees

• Adobe Systems Inc

Inventors

• Sachs Ian
• Shen Xiaoyong
• Paris Sylvain
• Hertzmann Aaron
• Shechtman Elya
• Price Brian

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06K9/4604

  Physics · mapped topic