Method and system for graph-based panoptic segmentation

Patent metadata

Field	Value
Publication number	US-11816841-B2
Application number	US-202117204930-A
Country	US
Kind code	B2
Filing date	Mar 17, 2021
Priority date	Mar 17, 2021
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

In methods and systems for graph-based panoptic segmentation of point clouds, points of a point cloud are received with a semantic label from a first category. Further, a plurality of unified cluster feature vectors from a second category are received, each being extracted from a cluster of points in the point cloud. Nodes of a constructed graph represent the unified feature vectors, and edges indicate the relationship between pairs of nodes. The edges are represented as an adjacency matrix indicating the existence or absence of an edge between pairs of nodes. A graph convolutional neural network uses the graph to predict an instance label for each node or an attribute for each edge, wherein the attribute of each edge is used for assigning the instance label to each node.

First claim

The invention claimed is:

1. A method for graph-based panoptic segmentation, the method comprising:
receiving points of a point cloud with a semantic label from a first category;
receiving a plurality of unified cluster feature vectors from a second category, each unified cluster feature vector being extracted from a cluster of points in the point cloud;
constructing a graph comprising nodes and edges from the plurality of unified cluster feature vectors, each node of the graph being one of the plurality of the unified feature vectors, each edge of the graph indicating the relationship between a pair of nodes of the graph and being represented as an adjacency matrix, wherein the adjacency matrix indicates the existence, or the lack of existence, of an edge between every two nodes;
feeding the nodes and the adjacency matrix to a graph convolutional neural network configured for predicting an instance label for each node or an attribute for each edge, wherein the attribute of each edge is used for assigning the instance label to each node; and
combining points with semantic labels for the first category and points with instance labels for the second category to generate points with both a semantic label and an instance label,
wherein each unified cluster feature vector is extracted from a plurality of points of a point cloud using at least one of a learnable sparse convolution operation and a PointNet model, which maps the plurality of points of the cluster to a 1×k vector, where k is a hyperparameter, and
wherein the unified cluster feature vector includes a centroid value of each cluster, generating a unified cluster feature vector of size 1×(k+3).

2. The method of claim 1 , wherein elements of the adjacency matrix are determined using at least one similarity distance between every two nodes.

3. The method of claim 2 , wherein the at least one similarity distance is a cosine similarity and a Euclidean distance between.

4. The method of claim 3 , wherein the elements of the adjacency matrix are determined using a criterion, the criterion being the edge exist if the cosine similarity between two nodes is greater than a prespecified threshold and the Euclidean distance between the two nodes is less than another prespecified threshold.

5. The method of claim 4 , wherein the graph convolutional neural network is configured for node classification to predict an instance label for each node of the graph, each point of the point cloud being labelled with the instance label of its respective node's instance label.

6. The method of claim 3 , wherein the graph convolutional neural network is configured for edge classification to predict the attribute for the edge between every two nodes, the nodes of the graph connected together by at least one edge being assigned an instance label, each point of the point cloud being labelled with the instance label of its respective node's instance label.

7. The method of claim 1 , wherein each point of the point cloud comprises at least spatial coordinates and a semantic label of the point.

8. The method of claim 1 , wherein the plurality of clusters are determined using at least one of k-means clustering, partition around medoids clustering, and density-based clustering (DBSCAN).

9. A system for graph-based panoptic segmentation using a graph convolutional neural network, comprising:
a memory storing instructions;
one or more processors coupled to the memory and configured to execute the instructions to:
receive points of a point cloud with a semantic label from a first category;
receive a plurality of unified cluster feature vectors from a second category, each unified cluster feature vector being extracted from a cluster of points in the point cloud;
construct a graph comprising nodes and edges from the plurality of unified cluster feature vectors, each node of the graph being the unified feature vector, each edge of the graph indicating the relationship between every two nodes of the graph and being represented as an adjacency matrix, wherein the adjacency matrix indicates the existence, or the lack of existence, of an edge between every two nodes;
feed the nodes and the adjacency matrix to a graph convolutional neural network configured for predicting an instance label for each node or an attribute for each edge, wherein the attribute of each edge is used for assigning the instance label to each node; and
combine points with semantic labels for the first category and points with instance labels for the second category to generate points with both a semantic label and an instance label,
wherein each unified cluster feature vector is extracted from a plurality of points of a point cloud using at least one of a learnable sparse convolution operation and a PointNet model, which maps the plurality of points the cluster to a 1×k vector, where k is a hyperparameter, and
wherein the unified cluster feature vector includes a centroid value of each cluster, generating a unified cluster feature vector of size 1×(k+3).

10. The system of claim 9 , wherein elements of the adjacency matrix are determined using at least one similarity distance between every two nodes.

11. The system of claim 10 , wherein the at least one similarity distance is a cosine similarity and a Euclidean distance between.

12. The system of claim 11 , wherein the elements of the adjacency matrix are determined using a criterion, the criterion being the edge exist if the cosine similarity between two nodes is greater than a prespecified threshold and the Euclidean distance between the two nodes is less than another prespecified threshold.

13. The system of claim 12 , wherein the graph convolutional neural network is configured for node classification to predict an instance label for each node of the graph, each point of the point cloud being labelled with the instance label of its respective node's instance label.

14. The system of claim 11 , wherein the graph convolutional neural network is configured for edge classification to predict the attribute for the edge between every two nodes, the nodes of the graph connected together by at least one edge being assigned an instance label, each point of the point cloud being labelled with the instance label of its respective node's instance label.

15. The system of claim 9 , wherein each point of the point cloud comprises at least spatial coordinates and a semantic label of the point.

16. The system of claim 9 , wherein the plurality of clusters are determined using at least one of k-means clustering, partition around medoids clustering, and density-based clustering (DBSCAN).

Assignees

• Cheng Ran
• Razani Ryan
• Liu Bingbing
• Huawei Tech Co Ltd

Inventors

• Cheng Ran
• Razani Ryan
• Liu Bingbing

Classifications

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V30/274

  Syntactic or semantic context, e.g. balancing · CPC title