System and method for unsupervised superpixel-driven instance segmentation of remote sensing image

Patent metadata

Field	Value
Publication number	US-12165329-B2
Application number	US-202217667523-A
Country	US
Kind code	B2
Filing date	Feb 8, 2022
Priority date	Feb 8, 2022
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

A system and method for unsupervised superpixel-driven instance segmentation of a remote sensing image are provided. The remote sensing image is divided into one or more image patches. The one or more image patches are processed to generate one or more superpixel aggregation patches based on a graph-based aggregation model, respectively. The graph-based aggregation model is configured to learn at least one of a spatial affinity or a feature affinity of a plurality of superpixels from each image patch and aggregate the plurality of superpixels based on the at least one of the spatial affinity or the feature affinity of the plurality of superpixels. The one or more superpixel aggregation patches are combined into an instance segmentation image.

First claim

What is claimed is:

1. A computer-implemented method for unsupervised superpixel-driven instance segmentation of a remote sensing image, comprising:
dividing, by a processor, the remote sensing image into one or more image patches;
processing, by the processor, the one or more image patches to generate one or more superpixel aggregation patches based on a graph-based aggregation model, respectively, at least by:
for each image patch from the one or more image patches,
processing the image patch to generate a corresponding superpixel patch that comprises a plurality of superpixels;
constructing a superpixel graph based on the image patch and the plurality of superpixels, wherein the superpixel graph comprises a plurality of nodes, with each node representing a corresponding superpixel from the plurality of superpixels;
determining multi-dimensional node features of each node based on multi-spectral values of pixels in the corresponding superpixel; and
feeding a superpixel node input comprising the multi-dimensional node features of each node into the graph-based aggregation model to generate a corresponding superpixel aggregation patch associated with the image patch; and

combining, by the processor, the one or more superpixel aggregation patches into an instance segmentation image.

2. The method of claim 1 , wherein
the graph-based aggregation model is configured to learn at least one of a spatial affinity or a feature affinity of the plurality of superpixels from each image patch and aggregate the plurality of superpixels based on the at least one of the spatial affinity or the feature affinity of the plurality of superpixels.

3. The method of claim 2 , further comprising:
constructing an adjacency matrix indicating a degree of adjacency among the plurality of nodes within at least one of a spatial space or a feature space, wherein the superpixel node input further comprises the adjacency matrix.

4. The method of claim 3 , wherein the superpixel node input further comprises a node degree of the superpixel graph.

5. The method of claim 3 , wherein constructing the adjacency matrix comprises:
for each element in the adjacency matrix,
determining a spatial adjacency factor for the element with respect to the spatial space;
determining a feature similarity factor for the element with respect to the feature space; and
determining a value for the element based on the spatial adjacency factor, the feature similarity factor, and an adjacency adjustment parameter.

6. The method of claim 5 , wherein the adjacency adjustment parameter is configured to adjust a balance of the spatial affinity and the feature affinity of the plurality of superpixels.

7. The method of claim 3 , wherein the feature space comprises a spectral space, and
the feature affinity comprises a spectral affinity of the plurality of superpixels.

8. The method of claim 3 , wherein the graph-based aggregation model comprises:
a backbone configured to generate global features and local features associated with the plurality of nodes based on the superpixel node input;
a fusion block configured to fuse the global features and the local features to generate fused features; and
a prediction block configured to assign the plurality of nodes to g node partitions based on the fused features, with g being a positive integer.

9. The method of claim 8 , wherein the g node partitions correspond to g aggregated groups of superpixels in the corresponding superpixel aggregation patch, respectively, and each aggregated group of superpixels comprise one or more superpixels from the plurality of superpixels, with the one or more superpixels corresponding to one or more nodes in a corresponding node partition.

10. The method of claim 9 , wherein g represents a total number of the aggregated groups of superpixels in the corresponding superpixel aggregation patch and is determined using a machine-learning-based detection method.

11. The method of claim 1 , wherein a loss function of the graph-based aggregation model comprises a normalized cut loss and a balanced cut loss, with the balanced cut loss being weighted by a loss adjustment parameter.

12. The method of claim 11 , wherein the loss adjustment parameter is configured to adjust a degree of importance of the balanced cut loss in the loss function.

13. The method of claim 1 , wherein each image patch has multi-spectral channels comprising a red band, a green band, a blue band, and a near-infrared band.

14. A system for unsupervised superpixel-driven instance segmentation of a remote sensing image, comprising:
a memory configured to store instructions; and
a processor coupled to the memory and configured to execute the instructions to perform a process comprising:
dividing the remote sensing image into one or more image patches;
processing the one or more image patches to generate one or more superpixel aggregation patches based on a graph-based aggregation model, respectively, at least by:

for each image patch from the one or more image patches,
processing the image patch to generate a corresponding superpixel patch that comprises a plurality of superpixels;
constructing a superpixel graph based on the image patch and the plurality of superpixels, wherein the superpixel graph comprises a plurality of nodes, with each node representing a corresponding superpixel from the plurality of superpixels;
determining multi-dimensional node features of each node based on multi-spectral values of pixels in the corresponding superpixel; and
feeding a superpixel node input comprising the multi-dimensional node features of each node into the graph-based aggregation model to generate a corresponding superpixel aggregation patch associated with the image patch; and

combining the one or more superpixel aggregation patches into an instance segmentation image.

15. The system of claim 14 , wherein
the graph-based aggregation model is configured to learn at least one of a spatial affinity or a feature affinity of the plurality of superpixels from each image patch and aggregate the plurality of superpixels based on the at least one of the spatial affinity or the feature affinity of the plurality of superpixels.

16. The system of claim 15 , wherein the processor is configured to perform the process further comprising:
constructing an adjacency matrix indicating a degree of adjacency among the plurality of nodes within at least one of a spatial space or a feature space;
wherein the superpixel node input further comprises the adjacency matrix.

17. The system of claim 16 , wherein the superpixel node input further comprises a node degree of the superpixel graph.

18. A non-transitory computer-readable storage medium configured to store instructions which, in response to an execution by a processor, cause the processor to perform a process comprising:
dividing a remote sensing image into one or more image patches;
processing the one or more image patches to generate one or more superpixel aggregation patches based on a graph-based aggregation model, respectively, at least by:
for each image patch from the one or more image patches,
processing the image patch to generate a corresponding superpixel patch that comprises a plurality of superpixels;
constructing a superpixel graph based on the image patch and the plurality of superpixels, wherein the superpixel graph comprises a plurality of nodes, with each node representing a corresponding superpixel from the plurality of superpixels;
determining multi-dimensional node features of each node b

Assignees

• Ping An Tech Shenzhen Co Ltd

Inventors

• Yang Zhicheng
• Zhou Hang
• Lai Jui-Hsin
• Han Mei

Classifications

• G06T2207/20072

  Graph-based image processing · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/187

  involving region growing; involving region merging; involving connected component labelling · CPC title

• G06T7/162

  involving graph-based methods · CPC title

• G06T7/136 Primary

  involving thresholding · CPC title