Generating image segmentation data using a multi-branch neural network

Patent metadata

Field	Value
Publication number	US-10614574-B2
Application number	US-201715784918-A
Country	US
Kind code	B2
Filing date	Oct 16, 2017
Priority date	Oct 16, 2017
Publication date	Apr 7, 2020
Grant date	Apr 7, 2020

Abstract

A multi-branch neural network generates segmentation data for a received image. The received image is provided to a high-level branch and a low-level branch. Based on the received image, the high-level branch generates a feature map of high-level image features, and the low-level branch generates a feature map of low-level image features. The high-level feature map and the low-level feature map are combined to generate a combined feature map. The combined feature map is provided to a boundary refinement module that includes a dense-connection neural network, which generates segmentation data for the received image, based on the combined feature map.

First claim

What is claimed is:

1. A method of producing segmentation data for related regions of an image, the method comprising:
receiving, by a multi-branch neural network, a graphical digital image, the multi-branch neural network comprising a high-level encoder neural network, a low-level encoder neural network, and a boundary refinement module;
providing the image to the high-level encoder neural network and to the low-level encoder neural network;
generating, by the high-level encoder neural network, a map of high-level features of the image, wherein the high-level features describe contextual qualities of the image;
generating, by the low-level encoder neural network, a map of low-level features of the image, wherein the low-level features describe fundamental qualities of the image;
combining the high-level feature map and the low-level feature map to form a combined feature map;
providing the combined feature map to a boundary refinement module having a dense-connection neural network including multiple densely connected units, wherein each densely connected unit of the multiple densely connected units receives the combined feature map as an input; and
receiving, from the boundary refinement module, segmentation data indicating a related region of the image.

2. The method of claim 1 , wherein the high-level feature map has a first spatial resolution smaller than a second spatial resolution of the low-level feature map.

3. The method of claim 1 , wherein the high-level encoder neural network has more layers than the low-level encoder neural network.

4. The method of claim 1 , wherein combining the high-level feature map and the low-level feature map comprises any of:
concatenating the high-level feature map and the low-level feature map,
convolving the high-level feature map and the low-level feature map,
performing a mathematical operation of the high-level feature map and the low-level feature map, or
analyzing, with an additional neural network, the high-level feature map and the low-level feature map.

5. The method of claim 4 , wherein, prior to the concatenating, the high-level feature map is resized to a same spatial resolution as the low-level feature map.

6. The method of claim 1 , wherein:
each densely connected unit of the multiple densely connected units is trained to determine boundaries of the related region of the image, and
the segmentation data is based on the boundaries determined by each of the multiple densely connected units.

7. The method of claim 1 , wherein the segmentation data includes a set of probabilities associated with a set of pixels in the image, wherein each probability in the set of probabilities indicates a likelihood that a respective associated pixel is included in the related region of the image.

8. A non-transitory computer-readable medium embodying program code for producing segmentation data for related regions of an image, the program code comprising instructions which, when executed by a processor, cause the processor to perform operations comprising:
receiving, by a multi-branch neural network, a graphical digital image, the multi-branch neural network comprising a high-level encoder neural network, a low-level encoder neural network, and a boundary refinement module;
providing the image to the high-level encoder neural network and to the low-level encoder neural network;
generating, by the high-level encoder neural network, a map of high-level features of the image, wherein the high-level features describe contextual qualities of the image;
generating, by the low-level encoder neural network, a map of low-level features of the image, wherein the low-level features describe fundamental qualities of the image;
combining the high-level feature map and the low-level feature map to form a combined feature map;
providing the combined feature map to a boundary refinement module having a dense-connection neural network including multiple densely connected units, wherein each densely connected unit of the multiple densely connected units receives the combined feature map as an input; and
receiving, from the boundary refinement module, segmentation data indicating a related region of the image.

9. The non-transitory computer-readable medium of claim 8 , wherein the high-level feature map has a first spatial resolution smaller than a second spatial resolution of the low-level feature map.

10. The non-transitory computer-readable medium of claim 8 , wherein the high-level encoder neural network has more layers than the low-level encoder neural network.

11. The non-transitory computer-readable medium of claim 8 , wherein combining the high-level feature map and the low-level feature map comprises any of:
concatenating the high-level feature map and the low-level feature map,
convolving the high-level feature map and the low-level feature map,
performing a mathematical operation of the high-level feature map and the low-level feature map, or
analyzing the high-level feature map and the low-level feature map with an additional neural network.

12. The non-transitory computer-readable medium of claim 8 , wherein:
each densely connected unit of the multiple densely connected units is trained to determine boundaries of the related region of the image, and
the segmentation data is based on the boundaries determined by each of the multiple densely connected units.

13. The non-transitory computer-readable medium of claim 8 , wherein the segmentation data includes a set of probabilities associated with a set of pixels in the image, wherein each probability in the set of probabilities indicates a likelihood that a respective associated pixel is included in the related region of the image.

14. A system for producing segmentation data for related regions of an image, the system comprising:
a means for receiving, by a multi-branch neural network, a graphical digital image, the multi-branch neural network comprising a high-level encoder neural network, a low-level encoder neural network, and a boundary refinement module;
a means for providing the image to the high-level encoder neural network and to the low-level encoder neural network;
a means for generating, by the high-level encoder neural network, a map of high-level features of the image, wherein the high-level features describe contextual qualities of the image;
a means for generating, by the low-level encoder neural network, a map of low-level features of the image, wherein the low-level features describe fundamental qualities of the image;
a means for combining the high-level feature map and the low-level feature map to form a combined feature map;
a means for providing the combined feature map to a boundary refinement module having a dense-connection neural network including multiple densely connected units, wherein each densely connected unit of the multiple densely connected units receives the combined feature map as an input; and
a means for receiving, from the boundary refinement module, segmentation data indicating a related region of the image.

15. The system of claim 14 , wherein the high-level feature map has a first spatial resolution smaller than a second spatial resolution of the low-level feature map.

16. The system of claim 14 , wherein the high-level encoder neural network has more layers than the low-level encoder neural network.

17. The system of claim 14 , wherein the means for combining the high-level feature map and the low-level feature map comprises any of:
concatenating the high-level feature map and the low-level feature map,
convolving

Assignees

• Adobe Inc

Inventors

• Zhang Jianming

Classifications

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T7/13

  Edge detection · CPC title

• G06T2207/20081

  Training; Learning · CPC title