Point cloud compression

Patent metadata

Field	Value
Publication number	US-10909725-B2
Application number	US-201816132230-A
Country	US
Kind code	B2
Filing date	Sep 14, 2018
Priority date	Sep 18, 2017
Publication date	Feb 2, 2021
Grant date	Feb 2, 2021

Abstract

A system comprises an encoder configured to compress attribute information and/or spatial for a point cloud and/or a decoder configured to decompress compressed attribute and/or spatial information for the point cloud. To compress the attribute and/or spatial information, the encoder is configured to convert a point cloud into an image based representation. Also, the decoder is configured to generate a decompressed point cloud based on an image based representation of a point cloud.

First claim

What is claimed is:

1. A system, comprising:
one or more sensors configured to capture a plurality of points that make up a point cloud, wherein respective ones of the points comprise spatial information for the point and attribute information for the point; and
an encoder configured to compress the point cloud, wherein to compress the point cloud, the encoder is configured to:
determine, for the point cloud, a plurality of patches each corresponding to portions of the point cloud, wherein each patch comprises points with surface normal vectors that deviate from one another less than a threshold amount;
for each patch,
generate a patch image comprising the set of points corresponding to the patch projected onto a patch plane; and
generate a patch image comprising depth information for the set of points corresponding to the patch, wherein the depth information represents depths of the points in a direction perpendicular to the patch plane;

pack generated patch images for each of the determined patches into one or more two dimensional (2D) image frames; and

encode the one or more 2D image frames.

2. The system of claim 1 , wherein the encoder is further configured to, for each patch:
generate a patch image comprising attribute information for one or more attributes of the set of points corresponding to the patch;
wherein said pack generated patch images for the determined patches into one or more 2D image frames comprises packing the patch image comprising attribute information.

3. The system of claim 1 , wherein the encoder is further configured to:
fill one or more spaces in the one or more 2D image frames not occupied by a patch image with a padding image.

4. The system of claim 3 , wherein the padding image is generated based, at least in part, on image blocks at one or more edges of the patch images, wherein the padding image is selected such that variances between the padding image and the image blocks are less than a threshold amount.

5. The system of claim 1 , wherein the encoder is further configured to:
generate an occupancy map indicating locations of patch images in the one or more 2D image frames; and
encode the occupancy map.

6. The system of claim 1 , wherein the one or more sensors are further configured to capture a plurality of points that make up the point cloud at a subsequent point in time subsequent to a first point in time, wherein at least some of the points of the point cloud have moved between the first point in time and the subsequent point in time,
wherein the encoder is further configured to:
determine vector movements of the at least some points between the first point in time and the subsequent point in time;
generate, for each patch comprising moved points, a patch image comprising vector movement information for at least some of the points, of the set of points corresponding to the patch, that have moved, wherein movement in a vector component direction for a point is represented by an intensity of a color at the point in the patch image.

7. The system of claim 1 , wherein the one or more 2D image frames are encoded in accordance with a high efficiency video coding (HEVC) standard.

8. A device, comprising:
a computer system configured to implement a decoder configured to:
receive one or more encoded two-dimensional (2D) image frames comprising patch images for a plurality of patches of a compressed point cloud, wherein, for each patch, the one or more encoded 2D image frames comprise: a patch image comprising a set of points of the patch projected onto a patch plane and a patch image comprising depth information for the set of points of the patch, wherein the depth information indicates depths of the points of the patch in a direction perpendicular to the patch plane;
decode the one or more 2D encoded image frames comprising the patch images;
determine, for each patch, spatial information for the set of points of the patch based, at least in part, on the patch image comprising the set of points of the patch projected onto the patch plane and the patch image comprising the depth information for the set of points of the patch; and
generate a decompressed version of the compressed point cloud based, at least in part, on the determined spatial information for the plurality of patches.

9. The device of claim 8 , wherein the one or more encoded 2D image frames are decoded in accordance with a high efficiency video coding (HEVC) standard.

10. The device of claim 8 , wherein to perform said determine, for each patch, spatial information for the set of points for the patch based, at least in part, on the patch image comprising the set of points of the patch projected onto the patch plane and the patch image comprising the depth information for the set of points of the patch, the decoder is further configured to:
identify the patch image comprising the set of points of the patch projected onto the patch plane;
identify the patch image comprising the depth information for the set of points of the patch, wherein the depth information is organized such that respective pieces of depth information for respective ones of the points of the set of points of the patch are located at locations in the depth patch image corresponding to locations of the respective ones of the points in the patch image comprising the set of points of the patch projected onto the patch plane; and
match respective pieces of depth information to respective points of the patch image comprising the set of points of the patch projected onto the patch plane.

11. The device of claim 10 , wherein the one or more encoded 2D image frames further comprise a patch image comprising attribute information for at least one of the patches,
wherein the decoder is further configured to:
identify the patch image comprising attribute information; and
assign attribute information included in the patch image to respective ones of the points of the set of points of the at least one patch,
wherein the attribute information included in the patch image is organized such that respective pieces of attribute information for respective ones of the points of the set of points of the at least one patch are located at locations in the patch image corresponding to locations of the respective ones of the points of the set of points of the patch as included in the patch image comprising the set of points of the patch projected onto the patch plane.

12. The device of claim 11 , wherein the encoder is further configured to:
receive an occupancy map indicating locations of respective patch images in the one or more encoded 2D image frames; and
utilize the received occupancy map to identify the respective patch images in the one or more encoded 2D image frames.

13. The device of claim 12 , wherein the occupancy map further indicates portions of the one or more encoded 2D image frames that do not include data for patch images, wherein the decoder is further configured to:
budget less decoding resources to decoding portions of the one or more encoded 2D image frames that do not include data for patch images than an amount of decoding resources budgeted to decode other portions of the one or more encoded 2D image frames that include data for patch images.

14. The device of claim 13 , wherein the one or more encoded 2D image frames received by the decoder are encoded as a plurality of blocks;
wherein the decoder is configured to, for each block of an image frame:
determine, based on the occupancy map, whether the block includes only data for a patch image, includes only non-patch image data, or includes a mix of data for a patch image and non-p

Assignees

• Apple Inc

Inventors

• Mammou Khaled
• Robinet Fabrice A
• Tourapis Alexandros
• Su Yeping

Classifications

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• H03M7/3059

  Digital compression and data reduction techniques where the original information is represented by a subset or similar information, e.g. lossy compression · CPC title

• G06T9/001 Primary

  Model-based coding, e.g. wire frame · CPC title

• H03M7/70

  Type of the data to be coded, other than image and sound · CPC title

• H03M7/30

  Compression (speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N); Expansion; Suppression of unnecessary data, e.g. redundancy reduction · CPC title