Image processing using coupled segmentation and edge learning

Patent metadata

Field	Value
Publication number	US-11790633-B2
Application number	US-202117365877-A
Country	US
Kind code	B2
Filing date	Jul 1, 2021
Priority date	Jul 1, 2021
Publication date	Oct 17, 2023
Grant date	Oct 17, 2023

Abstract

The disclosure provides a learning framework that unifies both semantic segmentation and semantic edge detection. A learnable recurrent message passing layer is disclosed where semantic edges are considered as explicitly learned gating signals to refine segmentation and improve dense prediction quality by finding compact structures for message paths. The disclosure includes a method for coupled segmentation and edge learning. In one example, the method includes: (1) receiving an input image, (2) generating, from the input image, a semantic feature map, an affinity map, and a semantic edge map from a single backbone network of a convolutional neural network (CNN), and (3) producing a refined semantic feature map by smoothing pixels of the semantic feature map using spatial propagation, and controlling the smoothing using both affinity values from the affinity map and edge values from the semantic edge map.

First claim

What is claimed is:

1. A processor for coupled segmentation and edge learning, comprising:
a backbone network configured to generate, from an input image, a semantic feature map, an affinity map, and a semantic edge map; and
a dynamic graph propagation (DGP) layer configured to produce a refined semantic feature map by smoothing pixels of the semantic feature map, wherein the DGP layer uses affinity values from the affinity map and edge values from the semantic edge map as a double-gate to control the smoothing.

2. The processor as recited in claim 1 , wherein the processor is further configured to generate a refined semantic edge map from the semantic edge map and the refined semantic edge map.

3. The processor as recited in claim 1 , wherein the DGP layer is configured to select, based on the affinity values, a single neighboring pixel for each of the pixels of the semantic feature map for the smoothing.

4. The processor as recited in claim 1 , wherein the DGP layer uses spatial propagation along multiple directions for the smoothing.

5. The processor as recited in claim 1 , wherein the semantic feature map includes multiple channels and the DGP layer encodes message passing paths of the pixels for the multiple channels.

6. The processor as recited in claim 5 , wherein each of the message passing paths for the multiple channels are independent of the other message passing paths for the other multiple channels.

7. The processor as recited in claim 1 , wherein the backbone network uses atrous spatial pyramid pooling layers to capture segmentation information from the input image and encodes the segmentation information to generate the semantic feature map.

8. The processor as recited in claim 1 , wherein the backbone network provides inputs to an edge classifier to generate the semantic edge map and generates one of the inputs by applying a sigmoid function on an output of a residual block of the backbone network having a resolution that is one eighth of the input image.

9. The processor as recited in claim 1 , where the processor is a convolutional neural network and the DGP layer is a learnable recurrent message passing layer.

10. A method for coupled segmentation and edge learning, comprising:
receiving an input image;
generating, from the input image, a semantic feature map, an affinity map, and a semantic edge map from a single backbone network of a convolutional neural network (CNN); and
producing a refined semantic feature map by smoothing pixels of the semantic feature map using spatial propagation, and controlling the smoothing using both affinity values from the affinity map and edge values from the semantic edge map as a double-gate.

11. The method as recited in claim 10 , further comprising generating a refined semantic edge map from the semantic edge map and the refined semantic edge map.

12. The method as recited in claim 10 , wherein the producing includes selecting based on the affinity values, a single neighboring pixel for each of the pixels of the semantic feature map for the smoothing.

13. The method as recited in claim 10 , wherein the spatial propagation includes propagating from four directions for the pixels of the semantic feature map.

14. The method as recited in claim 10 , wherein the producing includes using a learnable recurrent message passing layer that uses the semantic feature map, the affinity map, and the semantic edge map as input to produce the refined semantic feature map.

15. The method as recited in claim 10 , wherein generating the semantic feature map includes encoding segmentation information of the input image, generating the affinity map includes aggregating cross-layer information of the backbone network and encoding an affinity of pairwise neighboring pixels of the input image, and generating the semantic edge map includes combining dense skip-layer features with abundant details with edge classification layer through shared concatenation.

16. A system, comprising:
an image processor configured to perform a visual task based on at least one refined semantic map from an input image; and
a coupled segmentation and edge learning (CSEL) processor configured to produce the at least one refined semantic map by smoothing pixels of a semantic feature map using a dynamic graph propagation (DGP) layer, wherein the DGP layer is a learnable recurrent message passing layer that uses both affinity values and edge values extracted from the input image as a double-gate to control the smoothing.

17. The system as recited in claim 16 , wherein the at least one refined semantic map is a refined semantic feature map or a refined semantic edge map.

18. The system as recited in claim 16 , wherein the CSEL processor is configured to produce both a refined semantic feature map and a refined semantic edge map.

Assignees

• Nvidia Corp

Inventors

• Yu Zhiding
• Huang Rui
• Byeon Wonmin
• Liu Sifei
• Liu Guilin
• Breuel Thomas
• Anandkumar Anima
• Kautz Jan

Classifications

• G06V10/50 Primary

  by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06T7/13

  Edge detection · CPC title

• G06V10/758

  Involving statistics of pixels or of feature values, e.g. histogram matching · CPC title