Learning copy space using regression and segmentation neural networks

Patent metadata

Field	Value
Publication number	US-10970599-B2
Application number	US-201816191724-A
Country	US
Kind code	B2
Filing date	Nov 15, 2018
Priority date	Nov 15, 2018
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

Techniques are disclosed for characterizing and defining the location of a copy space in an image. A methodology implementing the techniques according to an embodiment includes applying a regression convolutional neural network (CNN) to an image. The regression CNN is configured to predict properties of the copy space such as size and type (natural or manufactured). The prediction is conditioned on a determination of the presence of the copy space in the image. The method further includes applying a segmentation CNN to the image. The segmentation CNN is configured to generate one or more pixel-level masks to define the location of copy spaces in the image, whether natural or manufactured, or to define the location of a background region of the image. The segmentation CNN may include a first stage comprising convolutional layers and a second stage comprising pairs of boundary refinement layers and bilinear up-sampling layers.

First claim

What is claimed is:

1. A method for learning copy space of an image, the method comprising:
applying, by a processor-based system, a regression convolutional neural network (CNN) to an image, the regression CNN to make a prediction of properties of a copy space, the properties including size and type, the prediction conditioned on a determination by the regression CNN of a copy space presence in the image;
applying, by the processor-based system, a segmentation CNN to the image, the segmentation CNN to generate one or more masks associated with locations of one or more of a manufactured copy space in the image, a natural copy space in the image, and a background region of the image;
searching a database of images for an image that includes a copy space associated with desired properties, the searching based on results from the regression CNN and the segmentation CNN; and
identifying at least one image that includes the copy space associated with the desired properties.

2. The method of claim 1 , wherein the segmentation CNN operates independently from the regression CNN.

3. The method of claim 1 , wherein the size is represented as a confidence score ranging from small to large, and the type is represented as a confidence score ranging from natural to manufactured.

4. The method of claim 1 , wherein the segmentation CNN includes a first stage comprising convolutional layers and a second stage comprising pairs of boundary refinement layers and bilinear up-sampling layers.

5. The method of claim 1 ,
wherein the regression CNN and the segmentation CNN are trained on a first set of annotated images and a second set of annotated images, the first set of annotated images including copy spaces, the second set of annotated images not including copy spaces; and
wherein at least one of the first and second sets of annotated images includes annotations indicating one or more of copy space presence, copy space size, copy space type, and pixel classifications to indicate one or more of background, natural, and manufactured classifications.

6. The method of claim 1 , wherein the image is a frame of a video.

7. A system for learning copy space of an image, the system comprising:
one or more processors;
a regression convolutional neural network (CNN) module at least one of controllable and executable by the one or more processors, and configured to operate on an image and make a prediction of properties of a copy space, the properties including size and type, the prediction conditioned on a determination by the regression CNN of a copy space presence in the image; and
a segmentation CNN module at least one of controllable and executable by the one or more processors, and configured to operate on the image and generate one or more masks associated with locations of one or more of a manufactured copy space in the image, a natural copy space in the image, and a background region of the image;
wherein the one or more processors are configured to search a database of images for an image that includes a copy space associated with desired properties, the one or more processors being configured to search based on results from the regression CNN and the segmentation CNN; and
wherein the one or more processors are further configured to identify at least one image that includes the copy space associated with the desired properties.

8. The system of claim 7 , wherein the segmentation CNN operates independently from the regression CNN.

9. The system of claim 7 , wherein the size is represented as a confidence score ranging from small to large, and the type is represented as a confidence score ranging from natural to manufactured.

10. The system of claim 7 , wherein the segmentation CNN includes a first stage comprising convolutional layers and a second stage comprising pairs of boundary refinement layers and bilinear up-sampling layers.

11. The system of claim 7 , wherein the regression CNN and the segmentation CNN are trained on annotated images.

12. The system of claim 7 , wherein the image is a frame of a video.

13. The system of claim 7 , wherein:
the regression CNN and the segmentation CNN are trained on a first set of annotated images and a second set of annotated images, the first set of annotated images including copy spaces, the second set of annotated images not including copy spaces; and
at least one of the first and second sets of annotated images includes annotations indicating one or more of copy space presence, copy space size, copy space type, and pixel classifications to indicate one or more of background, natural, and manufactured classifications.

14. A non-transitory computer program product having instructions encoded thereon that when executed by one or more computer processors cause the one or more computer processors to perform a process comprising:
applying a regression convolutional neural network (CNN) to an image, the regression CNN to make a prediction of properties of a copy space, the properties including size and type, the prediction conditioned on a determination by the regression CNN of a copy space presence in the image; and
applying a segmentation CNN to the image, the segmentation CNN to generate one or more masks associated with locations of one or more of a manufactured copy space in the image, a natural copy space in the image, and a background region of the image;
wherein the regression CNN and the segmentation CNN are trained on a first set of annotated images and a second set of annotated images, the first set of annotated images including copy spaces, the second set of annotated images not including copy spaces; and
wherein at least one of the first and second sets of annotated images includes annotations indicating one or more of copy space presence, copy space size, copy space type, and pixel classifications to indicate one or more of background, natural, and manufactured classifications.

15. The non-transitory computer program product of claim 14 , wherein the segmentation CNN operates independently from the regression CNN.

16. The non-transitory computer program product of claim 14 , wherein the size is represented as a confidence score ranging from small to large, and the type is represented as a confidence score ranging from natural to manufactured.

17. The non-transitory computer program product of claim 14 , wherein the segmentation CNN includes a first stage comprising convolutional layers and a second stage comprising pairs of boundary refinement layers and bilinear up-sampling layers.

18. The non-transitory computer program product of claim 14 , the process further comprising:
searching a database of images for an image that includes a copy space associated with desired properties, the searching based on results from the regression CNN and the segmentation CNN; and
identifying at least one image that includes the copy space associated with the desired properties.

19. The non-transitory computer program product of claim 14 , wherein the image is a frame of a video.

Assignees

• Adobe Inc

Inventors

• Ling Mingyang
• Filipkowski Alex
• Lin Zhe
• Zhang Jianming
• Gulati Samarth

Classifications

• G06V10/267

  by performing operations on regions, e.g. growing, shrinking or watersheds · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title