Point cloud compression with supplemental information messages

Patent metadata

Field	Value
Publication number	US-11711544-B2
Application number	US-202016912617-A
Country	US
Kind code	B2
Filing date	Jun 25, 2020
Priority date	Jul 2, 2019
Publication date	Jul 25, 2023
Grant date	Jul 25, 2023

Abstract

A system comprises an encoder configured to compress attribute information and/or spatial for a point cloud and/or a decoder configured to decompress compressed attribute and/or spatial information for the point cloud. To compress the attribute and/or spatial information, the encoder is configured to convert a point cloud into an image based representation. Also, the decoder is configured to generate a decompressed point cloud based on an image based representation of a point cloud. Additionally, an encoder is configured to signal and/or a decoder is configured to receive a supplementary message comprising volumetric tiling information that maps portions of 2D image representations to objects in the point. In some embodiments, characteristics of the object may additionally be signaled using the supplementary message or additional supplementary messages.

First claim

What is claimed is:

1. A non-transitory computer-readable medium storing program instructions, that when executed on or across one or more processors, cause the one or more processors to:
receive one or more video encoded image frames comprising attribute patch images and depth patch images packed into one or more 2D image frames;
receive an occupancy map indicating portions of the 2D image frames that correspond to attribute patch images and the depth patch images;
receive one or more supplementary messages comprising one or more entries, wherein the one or more entries comprise:
a first entry identifying a set of 2D coordinates representing portions of the occupancy map that correspond to the attribute patch images and the depth patch images that correspond to an object in a 3D scene;
a second entry comprising information identifying the attribute patch images and the depth patch images corresponding to the object that are comprised in the portions of the occupancy map represented by the 2D coordinates, wherein the identified attribute patch images are a sub-set of attribute patch images that is smaller than a set of attribute patch images represented in the occupancy map as a whole, and wherein the identified depth patch images are a sub-set of depth patch images that is smaller than a set of depth patch images represented in the occupancy map as a whole;
a third entry comprising information indicating a 3D volume space in the 3D scene corresponding to a reconstruction location for the object encoded using the sub-set of the attribute patch images and the sub-set of the depth patch images;
an additional entry comprising an object ID for the object, wherein the first entry, the second entry, and the third entry are linked to the object ID; and

selectively reconstruct, based on the supplementary message, the object, wherein said selectively reconstructing the object comprises identifying the relevant subset of attribute patch images and the relevant subset of depth patch images of the one or more 2D image frames corresponding to the object and reconstructing the object in the 3D volume space in the 3D scene.

2. The non-transitory computer-readable medium of claim 1 , wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to:
receive an additional supplementary message comprising an entry indicating a change in the 3D volume space in the 3D scene corresponding to the object from the 3D volume space in the 3D scene indicated in the supplementary message previously received;
determine a relationship between the entry of the additional supplementary message and at least one of the one or more entries of the supplementary message previously received; and
selectively reconstruct an updated version of the object in the 3D scene, wherein said selectively reconstructing comprises associating the relevant attribute patch images and depth patch images of the one or more 2D image frames identified in the supplementary message previously received with the changed 3D volume space indicated in the additional supplementary message.

3. The non-transitory computer-readable medium of claim 1 , wherein the program instructions, when executed on or across one or more processors, cause the one or more processors to:
receive auxiliary information indicating relationship information for the attribute patch images and the depth patch images; and

wherein the one or more supplementary messages comprise:
a fourth entry comprising information identifying portions of the auxiliary information corresponding to the object,
wherein the fourth entry is linked to the object ID.

4. The non-transitory computer-readable medium of claim 1 , wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to:
receive another supplementary message comprising:
an entry comprising information identifying different attribute patch images and different depth patch images linked to the object ID and/or information identifying portions of an occupancy map that correspond to the different attribute patch images and the different depth patch images; or
an entry comprising information defining another 3D volume space in the 3D scene linked to the object ID; and

selectively update a reconstructed version of the object associated with the object ID, such that the object associated with the object ID is rendered using different attribute and/or depth information included in the different attribute patch images and different depth patch images indicated in the other supplementary message or such that the object associated with the object ID is rendered at the other 3D volume space of the 3D scene indicated in the other supplementary message.

5. The non-transitory computer readable-medium of claim 1 , wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to:
receive an additional supplementary message comprising:
another entry comprising information identifying different attribute patch images and different depth patch images for another object linked to another object ID and information identifying portions of an occupancy map that correspond to the different attribute patch images and the different depth patch images for the other object linked to the other object ID;
an entry comprising information indicating a 3D volume space in the 3D scene for the other object linked to the other object ID; and
an object label assigned to the object ID, wherein the object linked to the object ID and the other object linked to the other object ID are assigned the same label; and

associate one or more object characteristics associate with the object label to both the object linked to the object ID and the other object linked to the other object ID.

6. The non-transitory computer readable-medium of claim 1 , wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to:
receive an additional supplementary message comprising another object label linked to the object ID, such that more than one object label is linked to the same attribute patch images and depth patch images via the object ID.

7. The non-transitory computer readable-medium of claim 6 , wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to:
receive another supplementary message indicating that the object label or the other object label assigned to the object ID is to be canceled; and
dissociate the cancelled one of the object label or the other object label from the attribute patch images and depth patch images or the 3D volume space associated with the object ID, without dissociating the one of the object labels that remains un-canceled.

8. The non-transitory computer-readable medium of claim 1 , wherein the one or more supplementary messages further comprise:
a flag indicating whether the object is visible or hidden;
a rendering priority for the object;
a dependency flag, indicating that rendering the object is dependent upon another object being rendered;
a collision shape for the object; or
a material ID for the object,

wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to:
render or refrain from rending the object based on the flag indicating whether the object is hidden or visible;
render the object in an order determined based on the rendering priority for the object;
determine whether to render the object based on its dependency with another object;
render the object according to the collision shape fo

Assignees

• Apple Inc

Inventors

• Tourapis Alexandros
• Kim Jungsun
• Mammou Khaled
• Singer David W
• Robinet Fabrice A

Classifications

• H04N19/597 Primary

  specially adapted for multi-view video sequence encoding · CPC title

• G06T7/10

  Segmentation; Edge detection (motion-based segmentation G06T7/215) · CPC title

• H04N19/20 Primary

  using video object coding · CPC title

• H04N19/467

  characterised by the embedded information being invisible, e.g. watermarking · CPC title

• H04N21/8193

  dedicated tools, e.g. video decoder software or IPMP tool · CPC title