Method and system for deep neural networks using dynamically selected feature-relevant points from a point cloud

Patent metadata

Field	Value
Publication number	US-11676005-B2
Application number	US-201816191011-A
Country	US
Kind code	B2
Filing date	Nov 14, 2018
Priority date	Nov 14, 2018
Publication date	Jun 13, 2023
Grant date	Jun 13, 2023

Abstract

Methods and systems for deep neural networks using dynamically selected feature-relevant points from a point cloud are described. A plurality of multidimensional feature vectors arranged in a point-feature matrix are received. Each row of the point-feature matrix corresponds to a respective one of the multidimensional feature vectors, and each column of the point-feature matrix corresponds to a respective feature. Each multidimensional feature vector represents a respective unordered point from a point cloud and each multidimensional feature vector includes a respective plurality of feature-correlated values, each feature-correlated value represents a correlation extent of the respective feature. A reduced-max matrix having a selected plurality of feature-relevant vectors is generated. The feature-relevant vectors are selected by, for each respective feature, identifying a respective multidimensional feature vector in the point-feature matrix having a maximum feature-correlated value associated with the respective feature. The reduced-max matrix is output to at least one neural network layer.

First claim

The invention claimed is:

1. A method performed by one or more processors, the method comprising:
executing a sequence of two or more feature extraction subsystems, execution of each feature extraction subsystem including:
processing a set of data points using a neural network layer to generate a plurality of multidimensional feature vectors arranged in a point-feature matrix having a plurality of rows and columns, each row of the point-feature matrix corresponding to a respective one of the multidimensional feature vectors, and each column of the point-feature matrix corresponding to a respective feature, each multidimensional feature vector representing a respective data point of the set of data points and including a respective plurality of feature-correlated values, each feature-correlated value representing a correlation extent of the respective feature;
processing the point-feature matrix using a critical point layer (CPL) to generate a reduced-max matrix having a selected plurality of feature-relevant vectors that is fewer in number than the plurality of multidimensional feature vectors in the point-feature matrix by:
identifying, for each column of the point-feature matrix, a multidimensional feature vector from the plurality of multidimensional feature vectors having a highest value; and
selecting, from a plurality of identified multidimensional feature vectors, the selected plurality of feature-relevant vectors to include in the reduced-max matrix;

wherein each row of the reduced-max matrix corresponds to a respective one of the feature-relevant vectors and each column of the reduced-max matrix corresponds to the respective feature of the point-feature matrix; and
outputting the reduced-max matrix;

wherein executing a given one feature extraction subsystem further includes:
providing the reduced-max matrix outputted by the given one feature extraction subsystem as the set of data points inputted to a subsequent feature extraction subsystem in the sequence of two or more feature extraction subsystems.

2. The method of claim 1 , wherein the selecting comprises:
generating an index vector containing row indices of the plurality of identified multidimensional feature vectors;
generating a sampled index vector by sampling the row indices in the index vector to a desired number; and
generating the reduced-max matrix using the row indices contained in the sampled index vector.

3. The method of claim 2 , wherein the sampling is deterministic, the method further comprising:
prior to the sampling, sorting the row indices contained in the index vector in an ascending order.

4. The method of claim 2 , wherein the desired number is predefined for performing batch processing for different respective point clouds.

5. The method of claim 2 , further comprising:
prior to the sampling, removing any repetitions in the row indices contained in the index vector.

6. The method of claim 1 , wherein reduced-max matrix outputted by a last feature extraction subsystem in the sequence of two or more feature extraction subsystems is outputted for processing by a final layer of a neural network by:
providing the reduced-max matrix, outputted by the last feature extraction subsystem, as input to the final layer, the final layer being executed by the one or more processors to extract a desired number of represented features from each feature-relevant vector; and
providing the output of the final layer to an object classification subsystem of the neural network to classify the selected plurality of feature-relevant vectors.

7. The method of claim 1 , further comprising:
receiving a plurality of unordered data points of a point cloud;
generating a plurality of transformed data by applying preliminary spatial transformation and filtering to the received unordered data points; and
providing the plurality of transformed data as the set of data points inputted to a first feature extraction subsystem in the sequence of two or more feature extraction subsystems.

8. The method of claim 7 , wherein the plurality of unordered data points are captured by a LIDAR sensor or by a red green blue-depth (RGB-D) camera.

9. A method performed by one or more processors to implement a neural network, the method comprising:
receiving a plurality of unordered data points of a point cloud;
providing the plurality of unordered data points as a set of data points to be processed using a sequence of two or more feature extraction subsystems of the neural network, wherein execution each feature extraction subsystem includes:
processing the set of data points using a neural network layer to generate a plurality of multidimensional feature vectors arranged in a point-feature matrix having a plurality of rows and columns, each row of the point-feature matrix corresponding to a respective one of the multidimensional feature vectors, and each column of the point-feature matrix corresponding to a respective feature, the respective multidimensional feature vector represents a respective data point of the set of data points and includes a plurality of feature-correlated values each representing correlation extent of the respective feature;
processing the point-feature matrix using a critical point layer (CPL) to:
generate a reduced-max matrix having a selected plurality of feature-relevant vectors that is fewer in number than the plurality of multidimensional feature vectors in the point-feature matrix by:
identifying, for each column of the point-feature matrix, a multidimensional feature vector from the plurality of multidimensional feature vectors having a highest value; and
selecting, from a plurality of identified multidimensional feature vectors, the selected plurality of feature-relevant vectors to include in the reduced-max matrix;

wherein each row of the reduced-max matrix corresponds to a respective one of the feature-relevant vectors and each column of the reduced-max matrix corresponds to the respective feature of the point-feature matrix; and
output the reduced-max matrix;

wherein executing a given one feature extraction subsystem further includes:
providing the reduced-max matrix outputted by the given one feature extraction subsystem as the set of data points inputted to a subsequent feature extraction subsystem in the sequence of two or more feature extraction subsystems;

processing the reduced-max matrix outputted by a final feature extraction subsystem of the sequence of two or more feature extraction subsystems using a final layer of the neural network; and
processing output of the final layer by an object classification subsystem of the neural network to obtain a plurality of classified points.

10. The method of claim 9 , wherein the CPL is applied to perform the selecting by:
generating an index vector containing row indices of the plurality of identified multidimensional feature vectors;
generating a sampled index vector by sampling the row indices in the index vector to a desired number; and
generating the reduced-max matrix using the row indices contained in the sampled index vector.

11. The method of claim 10 , sampling the row indices in the index vector to a desired number is deterministic sampling, and the CPL is further applied to:
prior to the sampling, sort the row indices contained in the index vector in an ascending order.

12. The method of claim 10 , wherein the desired number is predefined for performing batch processing for different respective point clouds.

13. The method of claim 10 , wherein the CPL is further applied to:
prior to the sampling, removing any repetitions in the row indices contained in the index vect

Assignees

• Nezhadarya Ehsan
• Taghavi Ehsan
• Liu Bingbing
• Huawei Tech Co Ltd

Inventors

• Nezhadarya Ehsan
• Taghavi Ehsan
• Liu Bingbing

Classifications

• G06V10/771

  Feature selection, e.g. selecting representative features from a multi-dimensional feature space · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2113

  by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title