Method, device and system of point cloud compression for intelligent cooperative perception system

Patent metadata

Field	Value
Publication number	US-12511787-B2
Application number	US-202217734858-A
Country	US
Kind code	B2
Filing date	May 2, 2022
Priority date	May 6, 2021
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

A method for point cloud compression of an intelligent cooperative perception (iCOOPER) for autonomous air vehicles (AAVs) includes: receiving a sequence of consecutive point clouds; identifying a key point cloud (K-frame) from the sequence of consecutive point clouds; transforming each of the other consecutive point clouds (P-frames) to have the same coordinate system as the K-frame; converting each of the K-frame and P-frames into a corresponding range image; spatially encoding the range image of the K-frame by fitting planes; and temporally encoding each of the range images of the P-frames using the fitting planes.

First claim

What is claimed is:

1 . A method of point cloud compression, selection, transmission and fusion for a distributed deep reinforcement learning-based intelligent cooperative perception (iCOOPER) for autonomous air vehicles (AAVs), comprising:
receiving, via an information-centric networking (ICN)-based communication, a sequence of consecutive point clouds, wherein the sequence of consecutive point clouds are sensory data named based on its region in an Octree structure with multiple resolutions; identifying a key point cloud (K-frame) from the sequence of consecutive point clouds; transforming each of the other consecutive point clouds (P-frames) to have the same coordinate system as the K-frame; converting each of the K-frame and P-frames into a corresponding range image; spatially encoding the range image of the K-frame by fitting planes; temporally encoding each of the range images of the P-frames using the fitting planes to obtain a sequence of compressed consecutive point clouds; determining, via a deep reinforcement learning-based adaptive transmission scheme, an optimal transmission policy for transmitting via the ICN-based communication the sequence of compressed consecutive point clouds based on information importance, location and trajectory, and wireless network state of the AAVs; decompressing the sequence of compressed consecutive point clouds and fusing the decompressed consecutive point clouds to compensate for network latency of the AAVs; and controlling the AAVs based on the fused consecutive point clouds, wherein controlling the AAVs includes analyzing the fused consecutive point clouds to detect objects; generating a global map with positions of the detected objects and AAVs position, orientation, and velocity state; computing a path and handling emergency events; and issuing commands to control the AAVs, wherein the method further comprises evaluating the point cloud compression including applying a recent iterative closest point (ICP)-based registration pipeline using a PCL (point cloud library) for registration; applying a second, CNN and PointNet-based approach for object detection; and applying a SqueezeSegV3, CNN-based approach for scene segmentation.

2 . The method of claim 1 , wherein the K-frame is a middle point cloud of the sequence of consecutive point clouds.

3 . The method of claim 1 , wherein:
the sequence of consecutive point clouds is generated by a LiDAR; and the sequence of compressed consecutive point clouds is obtained via Recurrent Neural Network (RNN)-based point cloud stream compression.

4 . The method of claim 1 , wherein the action of transforming each of the other consecutive point clouds (P-frames) to have the same coordinate system as the K-frame comprises transforming each of the P-frames to have the same coordinate system as the K-frame using inertial measurement unit (IMU) measurements.

5 . The method of claim 1 , wherein the action of transforming each of the other consecutive point clouds (P-frames) to have the same coordinate system as the K-frame comprises transforming each of the P-frames to have the same coordinate system as the K-frame using a six degree of freedom (DOF) transformation.

6 . The method of claim 1 , wherein the action of converting each of the K-frame and P-frames into a corresponding range image comprises converting each point in the same coordinate system to a pixel in the corresponding range image with a pixel value.

7 . The method of claim 1 , wherein the action of spatially encoding the range image of the K-frame by fitting planes comprises:
dividing the range image of the K-frame into unit tiles; and encoding the unit tiles using the fitting planes.

8 . A device of point cloud compression of an intelligent cooperative perception (iCOOPER) for autonomous air vehicles (AAVs) comprises a controller, the controller configured to:
receive, via an information-centric networking (ICN)-based communication, a sequence of consecutive point clouds, wherein the sequence of consecutive point clouds are sensory data named based on its region in an Octree structure with multiple resolutions; identify a key point cloud (K-frame) from the sequence of consecutive point clouds; transform each of the other consecutive point clouds (P-frames) to have the same coordinate system as the K-frame; convert each of the K-frame and P-frames into a corresponding range image; spatially encode the range image of the K-frame by fitting planes; temporally encode each of the range images of the P-frames using the fitting planes to obtain a sequence of compressed consecutive point clouds; determine, via a deep reinforcement learning-based adaptive transmission scheme, an optimal transmission policy for transmitting via the ICN-based communication the sequence of compressed consecutive point clouds based on information importance, location and trajectory, and wireless network state of the AAVs; decompress the sequence of compressed consecutive point clouds and fuse the decompressed consecutive point clouds to compensate for network latency of the AAVs; and control the AAVs based on the fused consecutive point clouds, wherein controlling the AAVs includes analyzing the fused consecutive point clouds to detect objects; generating a global map with positions of the detected objects and AAVs position, orientation, and velocity state; computing a path and handling emergency events; and issuing commands to control the AAVs, wherein the controller is further configured to evaluate the point cloud compression including applying a recent iterative closest point (ICP)-based registration pipeline using a PCL (point cloud library) for registration; applying a second, CNN and PointNet-based approach for object detection; and applying a SqueezeSegV3, CNN-based approach for scene segmentation.

9 . The device of claim 8 , wherein the K-frame is a middle point cloud of the sequence of consecutive point clouds.

10 . The device of claim 8 , wherein:
the sequence of consecutive point clouds is generated by a LiDAR; and the sequence of compressed consecutive point clouds is obtained via Recurrent Neural Network (RNN)-based point cloud stream compression.

11 . The device of claim 8 , wherein the action of transforming each of the other consecutive point clouds (P-frames) to have the same coordinate system as the K-frame comprises transforming each of the P-frames to have the same coordinate system as the K-frame using inertial measurement unit (IMU) measurements.

12 . The device of claim 8 , wherein the action of transforming each of the other consecutive point clouds (P-frames) to have the same coordinate system as the K-frame comprises transforming each of the P-frames to have the same coordinate system as the K-frame using a six degree of freedom (DOF) transformation.

13 . The device of claim 8 , wherein the action of converting each of the K-frame and P-frames into a corresponding range image comprises converting each point in the same coordinate system to a pixel in the corresponding range image with a pixel value.

14 . The device of claim 8 , wherein the action of spatially encoding the range image of the K-frame by fitting planes comprises:
dividing the range image of the K-frame into unit tiles; and encoding the unit tiles using the fitting planes.

15 . A system of point cloud compression of an intelligent cooperative perception (iCOOPER) for autonomous air vehicles (AAVs) comprises at least one AAV, the at least one AAV configured to
receive, via an information-centric networking (ICN)-based communication, a sequence of consecutiv

Assignees

• Intelligent Fusion Tech Inc

Inventors

• Zhao Qi
• Li Yi
• Tian Xin
• Chen Genshe
• Blasch Erik
• Pham Khanh

Classifications

• G06T3/00

  Geometric image transformations in the plane of the image · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G01S17/89 Primary

  for mapping or imaging · CPC title

• G06T9/001 Primary

  Model-based coding, e.g. wire frame · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title