Video-based point cloud compression with non-normative smoothing

Patent metadata

Field	Value
Publication number	US-11627314-B2
Application number	US-202017031792-A
Country	US
Kind code	B2
Filing date	Sep 24, 2020
Priority date	Sep 27, 2019
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

A system comprises an encoder configured to compress attribute and/or spatial information for a point cloud and/or a decoder configured to decompress compressed attribute and/or spatial information for the point cloud. To compress the attribute and/or spatial information, the encoder is configured to convert a point cloud into an image based representation. Also, the decoder is configured to generate a decompressed point cloud based on an image based representation of a point cloud. In some embodiments, an encoder applies an attribute transfer function to attributes of an image based representation with out-of-loop smoothing such that a decoder may apply any smoothing filter supported by the decoder without being restricted to using a similar smoothing filter as the encoder.

First claim

What is claimed is:

1. A non-transitory computer-readable medium storing program instructions that, when executed by one or more processors, cause the one or more processors to:
determine, for a three-dimensional (3D) scene or object, a plurality of patches, wherein each patch comprises points of the 3D scene or object projected on a patch plane;
pack attribute patch images and geometry patch images corresponding to the plurality of patches into corresponding attribute image frames and geometry image frames;
apply, at an encoder, an attribute transform function to the attribute patch images of the attribute image frames to compensate for distortion or artifacts to be introduced during a compression and a decompression of the geometry image frames;
perform, at the encoder, an out-of-loopsmoothing process to determine smoothing parameters to be applied at a smoothing filter of a decoder to smooth the geometry patch images of the geometry image frames, wherein, in the out-of-loop smoothing process, the attribute transform function is applied at the encoder to geometry image frames for which the smoothing parameters have not yet been applied; and
encode the geometry image frames and the attribute image frames that have had the attribute transform function applied.

2. The non-transitory computer-readable-medium of claim 1 , wherein the smoothing parameters comprise different smoothing parameters to be applied at the decoder for different layers of the 3D scene or object.

3. A non-transitory computer-readable medium storing program instructions that, when executed by one or more processors, cause the one or more processors to:
receive, at a decoder, an encoded bitstream comprising data for geometry images frames and data for attribute image frames of a three-dimensional (3D) object or scene, wherein the geometry image frames comprise geometry information for geometry patches of the 3D object or scene projected on patch planes and the attribute images frames comprise attribute information for attribute patches of the 3D object or scene projected on the patch planes;
decode the encoded bitstream;
reconstruct the geometry patches and the attribute patches for the 3D object or scene using decoded geometry image frames and decoded attribute image frames, decoded from the encoded bitstream;
apply out-of-loop smoothing parameters via a smoothing filter to:
the geometry patches of the geometry image frames; or
a reconstructed version of the 3D object or scene generated using the geometry patches of the geometry image frames;

determine parameters for an attribute transfer function to be applied to the attribute patches to compensate for changes in geometry due to the application of the out-of-loop smoothing parameters via the smoothing filter; and
apply the attribute transfer function to attribute values of the attribute patches to compensate for the changes in geometry of the smoothed geometry patches or the smoothed reconstructed version of the 3D object or scene, wherein the changes in geometry are relative to respective geometries of the geometry patches of the geometry images prior to applying the smoothing filter.

4. The non-transitory computer-readable medium of claim 3 , wherein the program instructions, further cause the one or more processors to:
apply an attribute smoothing filter to the attribute values of the attribute patches, wherein the attribute smoothing filter is applied in combination with applying the attribute transfer function.

5. The non-transitory computer-readable medium of claim 3 , wherein the smoothing filter is selected by a user or a designer of the decoder.

6. The non-transitory computer-readable medium of claim 3 , wherein:
the encoded bit-stream comprises smoothing parameters to be applied at the decoder, and
the out-of-loop smoothing parameters applied via the smoothing filter to the geometry patches or the reconstructed version of the 3D object or scene is selected or adjusted at the decoder based, at least in part, on the smoothing parameters signaled in the bitstream.

7. The non-transitory computer-readable medium of claim 3 , wherein the smoothing filter used by the decoder is a non-normative smoothing filter with regard to a compression standard used to compress and decompress the 3D object or scene.

8. The non-transitory computer-readable medium of claim 3 , wherein the program instructions further cause the one or more processors to:
select the out-of-loop smoothing parameters to be applied, via the smoothing filter at the decoder, from a plurality of smoothing filter parameters supported by the decoder based on an operation mode of a device comprising the decoder or based on a smoothing parameter signaled in the bitstream.

9. The non-transitory computer-readable medium of claim 8 , wherein the program instructions cause the one or more processors to:
select a first smoothing filter type if the device comprising the decoder is connected to an external power source; and
select a second smoothing filter type if the device comprising the decoder is receiving power from a battery of the device.

10. The non-transitory computer-readable medium of claim 3 , wherein the program instructions, cause the one or more processors to:
perform said apply the out-of-loop smoothing parameters, said determine parameters for the attribute transfer function, and said apply the attribute transfer function on a point-by-point basis in parallel for a plurality of points of the 3D object or scene.

11. The non-transitory computer-readable medium of claim 3 , wherein the program instructions, cause the one or more processors to:
perform said apply the out-of-loop smoothing parameters, said determine parameters for the attribute transfer function, and said apply the attribute transfer function as serial operations performed in series for a set of points of the 3D object or scene.

12. The non-transitory computer-readable medium of claim 3 , wherein the program instructions, cause the one or more processors to:
perform said apply the out-of-loop smoothing parameters, said determine parameters for the attribute transfer function, and said apply the attribute transfer function in a 2D domain on the geometry patch images and the attribute patch images for the 3D object or scene.

13. The non-transitory computer-readable medium of claim 3 , wherein the program instructions, cause the one or more processors to:
perform said apply the out-of-loop smoothing parameters, said determine parameters for the attribute transfer function, and said apply the attribute transfer function in a 3D domain on the reconstructed version of the 3D object or scene comprising a reconstructed geometry of the 3D object or scene.

14. The non-transitory computer-readable medium of claim 3 , wherein the program instructions, cause the one or more processors to:
perform said apply the out-of-loop smoothing parameters, said determine parameters for the attribute transfer function, and said apply the attribute transfer function as part of a rendering process for rendering the reconstructed version of the 3D object or scene.

15. The non-transitory computer-readable medium of claim 3 , wherein the program instructions cause the one or more processors to:
determine one or more layers of a plurality of layers of the 3D object or scene to reconstruct; and
perform said apply the out-of-loop smoothing parameters, said determine parameters for the attribute transfer function, and said apply the attribute transfer function only on the one or more layers determined to be reconstructed.

16. The non-transitory com

Assignees

• Apple Inc

Inventors

• Tourapis Alexandros
• Kim Jungsun
• Mammou Khaled
• Mehlem Dominik

Classifications

• H04N19/12

  Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264 · CPC title

• H04N19/80

  Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence) · CPC title

• H04N19/184

  the unit being bits, e.g. of the compressed video stream · CPC title

• H04N19/117 Primary

  Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) · CPC title

• H04N19/597 Primary

  specially adapted for multi-view video sequence encoding · CPC title