Point cloud data reformatting

Patent metadata

Field	Value
Publication number	US-11804009-B2
Application number	US-202117564119-A
Country	US
Kind code	B2
Filing date	Dec 28, 2021
Priority date	Apr 22, 2020
Publication date	Oct 31, 2023
Grant date	Oct 31, 2023

Abstract

Described herein are systems, methods, and computer readable media for performing data conversion on sensor data to obtain modified sensor data that is formatted/structured appropriately for downstream processes that rely on the sensor data as input. The sensor data can include point cloud data captured by a LiDAR, for example. A grid structure and corresponding grid characteristics can be determined and the sensor data can be converted to grid-based sensor data by associating the grid structure and its characteristics with the sensor data. Generating the grid-based sensor data can include reformatting the point cloud data to superimpose the grid structure and its grid characteristics onto the point cloud data. Various downstream processing that cannot feasibly be performed on the raw sensor data can then be performed efficiently on the modified grid-based sensor data by virtue of the grid structure imbuing the sensor data with spatial proximity information.

First claim

What is claimed is:

1. A computer-implemented method for performing data conversion of sensor data, the method comprising:
receiving the sensor data;
determining grid characteristics of a grid structure comprising a plurality of grid unit elements, the grid characteristics comprising a size and a granularity of the grid structure, wherein the determining of the grid characteristics comprises:
determining that greater than a threshold number of the grid unit elements have an absence of data points of the sensor data; and
modifying a granularity of the grid structure to decrease a number of the grid unit elements that have an absence of data points;

converting the sensor data to grid-based sensor data, the converting comprising superimposing the grid structure with the sensor data;
tracking an object represented by the grid-based sensor data based on only a subset of the grid-based sensor data, the tracking comprising:
determining a first set of grid elements corresponding to a first position of the object at a first time;
determining a second set of grid elements within a threshold distance of the first set of grid elements;
selectively searching the second set of grid elements to obtain a third set of grid elements, the third set of grid elements corresponding to a second position of the object at a second time; and
monitoring the third set of grid elements while refraining from monitoring a remainder of the grid elements besides the third set of grid elements while tracking a movement of the object at the second time, wherein the grid structure comprises a two-dimensional (2D) grid structure; and the conversion of the sensor data to grid-based sensor data comprises:
positioning corresponding data points of successive frames of the sensor data within a threshold distance of each other on the grid-based sensor data.

2. The computer-implemented method of claim 1 , wherein the sensor data comprises point cloud data; and the method further comprising:
discarding a portion of the point cloud data, wherein the portion does not correspond to the second set of grid elements.

3. The computer-implemented method of claim 1 , wherein the grid structure comprises a two-dimensional (2D) grid structure; and the conversion of the sensor data to grid-based sensor data comprises:
positioning corresponding data points of successive frames of the sensor data within a threshold distance of each other on the grid-based sensor data.

4. The computer-implemented method of claim 1 , wherein the grid characteristics are determined based on a distribution of the sensor data and are updated in response to a change in the sensor data.

5. The computer-implemented method of claim 1 , wherein the granularity indicates a corresponding physical region covered by a grid element; and the granularity satisfies a minimum coverage area of a geographic region.

6. The computer-implemented method of claim 1 , further comprising:
receiving a modification to the sensor data;
modifying the grid structure based on the modification to the sensor data, wherein the modified grid structure has a second granularity different from the granularity;
reformatting the sensor data according to the modified grid structure; and
superimposing the reformatted sensor data into the modified grid structure.

7. The computer-implemented method of claim 1 , further comprising:
determining whether the grid-based sensor data comprises more than a threshold proportion of empty grid elements having no sensor data superimposed within; and
in response to determining that the grid-based sensor data comprises more than a threshold proportion of empty grid elements having no sensor data superimposed within, increasing a number of grid elements in the grid structure until the proportion of empty grid elements is within the threshold proportion.

8. A system for performing data conversion of sensor data, the system comprising:
at least one processor; and
at least one memory storing computer-executable instructions, wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
receive the sensor data;
determine grid characteristics of a grid structure comprising a plurality of grid unit elements, the grid characteristics comprising a size and a granularity of the grid structure, wherein the determining of the grid characteristics comprises:
determining that greater than a threshold number of the grid unit elements have an absence of data points of the sensor data; and
modifying a granularity of the grid structure to decrease a number of the grid unit elements that have an absence of data points;

convert the sensor data to grid-based sensor data, the converting comprising superimposing the grid structure with the sensor data; and
track an object represented by the grid-based sensor data based on only a subset of the grid-based sensor data, the tracking comprising:
determining a first set of grid elements corresponding to a first position of the object at a first time;
determining a second set of grid elements within a threshold distance of the first set of grid elements;
selectively searching the second set of grid elements to obtain a third set of grid elements, the third set of grid elements corresponding to a second position of the object at a second time; and

monitoring the third set of grid elements while refraining from monitoring a remainder of the grid elements besides the third set of grid elements while tracking a movement of the object at the second time, wherein the grid structure comprises a two-dimensional (2D) grid structure; and the conversion of the sensor data to grid-based sensor data comprises:
positioning corresponding data points of successive frames of the sensor data within a threshold distance of each other on the grid-based sensor data.

9. The system of claim 8 , wherein the sensor data comprises point cloud data, and the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
discard a portion of the point cloud data, wherein the portion does not correspond to the second set of grid elements.

10. The system of claim 8 , wherein the grid structure comprises a two-dimensional (2D) grid structure; and the conversion of the sensor data to grid-based sensor data comprises:
positioning corresponding data points of successive frames of the sensor data within a threshold distance of each other on the grid-based sensor data.

11. The system of claim 8 , wherein the grid characteristics are determined based on a distribution of the sensor data and are updated in response to a change in the sensor data.

12. The system of claim 8 , wherein the granularity indicates a corresponding physical region covered by a grid element; and the granularity satisfies a minimum coverage area of a geographic region.

13. The system of claim 8 , wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
receive a modification to the sensor data;
modify the grid structure based on the modification to the sensor data, wherein the modified grid structure has a second granularity different from the granularity;
reformat the sensor data according to the modified grid structure; and
superimpose the reformatted sensor data into the modified grid structure.

14. The system of claim 8 , wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
determine whether the grid-based

Assignees

• Pony Ai Inc

Inventors

• Meng Pingfan
• Wu Michael
• Rush Brian James

Classifications

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title