Three-dimensional point cloud tracking apparatus and method using recurrent neural network

Patent metadata

Field	Value
Publication number	US-10705216-B2
Application number	US-201715834781-A
Country	US
Kind code	B2
Filing date	Dec 7, 2017
Priority date	Dec 7, 2017
Publication date	Jul 7, 2020
Grant date	Jul 7, 2020

Abstract

The embodiments of the present invention provide a three-dimensional point cloud tracking apparatus and method using a recurrent neural network. The three-dimensional point cloud tracking apparatus and method can track the three-dimensional point cloud of the entire environment and model the entire environment by using a recurrent neural network model. Therefore, the three-dimensional point cloud tracking apparatus and method can be used to reconstruct the three-dimensional point cloud of the entire environment at the current moment and also can be used to predict the three-dimensional point cloud of the entire environment at a later moment.

First claim

What is claimed is:

1. A three-dimensional point cloud tracking apparatus using a recurrent neural network, comprising:
an input/output interface, configured to receive different observed three-dimensional point clouds at different moments in an environment, wherein the observed three-dimensional point clouds are obtained by a scanning of at least one LiDAR;
a memory, configured to store at least one memory three-dimensional point cloud related to the environment; and
a processor, electrically connected to the input/output interface and the memory respectively and configured to receive the observed three-dimensional point clouds and the at least one memory three-dimensional point cloud, wherein when receiving the observed three-dimensional point cloud of the environment at a first moment, the processor utilizes at least one recurrent neural network model to perform an environment reconstruction operation on the observed three-dimensional point cloud and the at least one memory three-dimensional point cloud to obtain a reconstructed three-dimensional point cloud of the environment at the first moment, and then utilizes the recurrent neural network model again to perform an environment prediction operation on the at least one memory three-dimensional point cloud and a blank three-dimensional point cloud to obtain a predicted three-dimensional point cloud of the environment at a second moment, wherein the second moment is later than the first moment;
wherein the at least one memory three-dimensional point cloud comprises a first memory three-dimensional point cloud and a second memory three-dimensional point cloud, and the step of utilizing the recurrent neural network model to perform the environment reconstruction operation on the observed three-dimensional point cloud and the at least one memory three-dimensional point cloud to obtain the reconstructed three-dimensional point cloud of the environment at the first moment comprises:
performing a first sparse convolution operation on the observed three-dimensional point cloud to obtain a first calculated three-dimensional point cloud;
performing a second sparse convolution operation on the first calculated three-dimensional point cloud and the first memory three-dimensional point cloud to obtain a second calculated three-dimensional point cloud and updating the first memory three-dimensional point cloud with the second calculated three-dimensional point cloud; and
performing a third sparse convolution operation on the second calculated three-dimensional point cloud and the second memory three-dimensional point cloud to obtain the reconstructed three-dimensional point cloud of the environment at the first moment and updating the second memory three-dimensional point cloud with the reconstructed three-dimensional point cloud.

2. The three-dimensional point cloud tracking apparatus according to claim 1 , wherein the step of utilizing the recurrent neural network model again to perform the environment prediction operation on the at least one memory three-dimensional point cloud and the blank three-dimensional point cloud to obtain the predicted three-dimensional point cloud of the environment at the second moment comprises:
performing a fourth sparse convolution operation on the blank three-dimensional point cloud and the first memory three-dimensional point cloud to obtain a third calculated three-dimensional point cloud; and
performing a fifth sparse convolution operation on the third calculated three-dimensional point cloud and the second memory three-dimensional point cloud to obtain the predicted three-dimensional point cloud of the environment at the second moment.

3. The three-dimensional point cloud tracking apparatus according to claim 2 , wherein the processor is further configured to define at least one weight self-defined function, at least one first sparse convolution kernel and at least one second sparse kernel, and the step of updating the first memory three-dimensional point cloud with the second calculated three-dimensional point cloud further comprises:
utilizing the weight self-defined function to determine a weight vector from the first memory three-dimensional point cloud, the second calculated three-dimensional point cloud, the first sparse convolution kernel and the second sparse convolution kernel, and updating the first memory three-dimensional point cloud to a result of substituting the first memory three-dimensional point cloud, the second calculated three-dimensional point cloud and the weight vector into a weight formula.

4. The three-dimensional point cloud tracking apparatus according to claim 3 , wherein the step of updating the second memory three-dimensional point cloud with the reconstructed three-dimensional point cloud further comprises:
utilizing the weight self-defined function to determine the weight vector from the second memory three-dimensional point cloud, the reconstructed three-dimensional point cloud, the first sparse convolution kernel and the second sparse convolution kernel, and updating the second memory three-dimensional point cloud to a result of substituting the second memory three-dimensional point cloud, the reconstructed three-dimensional point cloud and the weight vector into the weight formula.

5. The three-dimensional point cloud tracking apparatus according to claim 4 , wherein the weight self-defined function, the first sparse convolution kernel and the second sparse convolution kernel are defined by the three-dimensional point cloud tracking apparatus after completing a training mode, and a component of the weight vector is located between 0 and 1.

6. The three-dimensional point cloud tracking apparatus according to claim 5 , wherein the weight formula is p×C 1 +(1−p)×C 2 , wherein p is the weight vector, and C 1 and C 2 respectively are the first memory three-dimensional point cloud and the second calculated three-dimensional point cloud, or the second memory three-dimensional point cloud and the reconstructed three-dimensional point cloud.

7. A three-dimensional point cloud tracking method using a recurrent neural network and executed in a three-dimensional point cloud tracking apparatus, the three-dimensional point cloud tracking apparatus comprising an input/output interface, a memory, and a processor, the three-dimensional point cloud tracking method comprising:
configuring the input/output interface to receive different observed three-dimensional point clouds at different moments in an environment, wherein the observed three-dimensional point clouds are obtained by a scanning of at least one LiDAR;
configuring the memory to store at least one memory three-dimensional point cloud related to the environment; and
configuring the processor to receive the observed three-dimensional point clouds and the at least one memory three-dimensional point cloud, configuring, when receiving the observed three-dimensional point cloud of the environment at a first moment, the processor to utilize at least one recurrent neural network model to perform an environment reconstruction operation on the observed three-dimensional point cloud and the at least one memory three-dimensional point cloud to obtain a reconstructed three-dimensional point cloud of the environment at the first moment, and configuring the processor to utilize the recurrent neural network model again to perform an environment prediction operation on the at least one memory three-dimensional point cloud and a blank three-dimensional point cloud to obtain a predicted three-dimensional point cloud of the environment at a second moment, wherein the second moment is later than the first moment;
wherein the at least one memory three-dimensional point cloud comprises a first memory three-dimensional point cloud and a second m

Assignees

• Inst Information Ind

Inventors

• Wang Chih-Wei
• Tsai Ywe-Ting

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title