Volumetric video with auxiliary patches

Patent metadata

Field	Value
Publication number	US-12101507-B2
Application number	US-202017786005-A
Country	US
Kind code	B2
Filing date	Dec 17, 2020
Priority date	Dec 19, 2019
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

Methods and devices for encoding and decoding data representative of a 3D scene are disclosed. A set of first patches is generated from a first MVD content acquired from a first region of the 3D scene. A patch is a part of one of the views of the MVD content. A set of second patches is generated from a second MVD content acquired from a second region of the 3D scene. An atlas packing first and second patches is generated and associated with metadata indicating, for a patch of the atlas, whether the patch is a first or a second patch At the decoding side, first patches are used for rendering the viewport image and second patches are used for pre-processing or post-processing the viewport image.

First claim

The invention claimed is:

1. A method for encoding a 3D scene in a data stream, the method comprising:
generating a set of first patches from a first multi-view-plus-depth (MVD) content acquired for a rendering of the 3D scene, a first patch being a part of one of the views of the first MVD content;
generating a set of second patches from a second MVD content, wherein the second patches are not intended to be used for view rendering and being acquired for a pre-processing or a post-processing of the rendering of the 3D scene use, a second patch being a part of one of the views of the second MVD content;
generating a first atlas packing said first patches and a second atlas packing said second patches, an atlas being an image packing patches according to an atlas layout; and
encoding said first and second atlases in said data stream.

2. The method of claim 1 , wherein the second MVD content is acquired in a resolution lower than a resolution of the first MVD content.

3. The method of claim 1 , wherein a patch is a part of one view of a MVD content obtained by removing information redundancy between views of the MVD content.

4. The method of claim 1 , wherein said first and second patches are packed in a common atlas, the common atlas being associated with metadata indicating, for each patch whether the patch is the first patch or the second patch.

5. The method of claim 1 , wherein the second atlas associated with metadata indicating that the second patches are not intended to be used for view rendering and comprising a list of types of pre-processing or a post-processing of the rendering of the 3D scene.

6. The method of claim 5 , wherein a type of pre-processing or a post-processing of the rendering of the 3D scene belongs to a group of types of pre-processing or a post-processing of the rendering of the 3D scene comprising collision detection, interactions, and scene relighting.

7. A method for decoding a 3D scene from a data stream, the method comprising:
decoding said data stream to retrieve a first atlas and a second atlas, wherein the first atlas and the second atlas are images packing first or second patches respectively according to an atlas layout, a first patch being a part of one view of a first MVD content acquired for a rendering of the 3D scene, a second patch being a part of one view of a second MVD content, being not intended to be used for view rendering and being acquired for a pre-processing or a post-processing of the rendering of the 3D scene;
using second patches for a pre-processing and/or a post-processing of the rendering of the 3D scene; and
rendering a viewport image from a point of view within the 3D scene by using first patches.

8. The method of claim 7 , wherein the second MVD content has a resolution lower than a resolution of the first MVD content.

9. The method of claim 7 , wherein the first atlas and the second atlas are a unique atlas, the unique atlas being associated with metadata indicating, for each patch whether the patch is the first patch or the second patch.

10. The method of claim 7 , wherein the second atlas associated with metadata indicating that the second patches are not intended to be used for view rendering and comprising a list of types of pre-processing or a post-processing of the rendering of the 3D scene.

11. The method of claim 10 , wherein a type of pre-processing or a post-processing of the rendering of the 3D scene belongs to a group of types of pre-processing or a post-processing of the rendering of the 3D scene comprising collision detection, interactions, and scene relighting.

12. A device for encoding a 3D scene in a data stream, the device comprising a memory associated with a processor configured for:
generating a set of first patches from a first multi-view-plus-depth (MVD) content acquired for a rendering of the 3D scene, a first patch being a part of one of the views of the first MVD content;
generating a set of second patches from a second MVD content, wherein the second patches are not intended to be used for view rendering and being acquired for a pre-processing or a post-processing of the rendering of the 3D scene, a second patch being a part of one of the views of the second MVD content;
generating a first atlas packing said first patches and a second atlas packing said second patches, an atlas being an image packing patches according to an atlas layout; and
encoding said first and second atlases in said data stream.

13. The device of claim 12 , wherein the second MVD content is acquired in a resolution lower than a resolution of the first MVD content.

14. The device of claim 12 , wherein a patch is a part of one view of a MVD content obtained by removing information redundancy between views of the MVD content.

15. The device of claim 12 , wherein said first and second patches are packed in a unique atlas, the unique atlas being associated with metadata indicating, for each patch whether the patch is the first patch or the second patch.

16. The device of claim 12 , wherein the second atlas associated with metadata indicating that the second patches are not intended to be used for view rendering and comprising a list of types of pre-processing or a post-processing of the rendering of the 3D scene.

17. The device of claim 16 , wherein a type of pre-processing or a post-processing of the rendering of the 3D scene belongs to a group of types of pre-processing or a post-processing of the rendering of the 3D scene comprising collision detection, interactions, and scene relighting.

18. A device for decoding a 3D scene from a data stream, the device comprising a processor configured for:
decoding said data stream to retrieve a first atlas and a second atlas, wherein the first atlas and the second atlas are images packing first or second patches respectively according to an atlas layout, a first patch being a part of one view of a first MVD content acquired for a rendering of the 3D scene, a second patch being a part of one view of a second MVD content, being not intended to be used for view rendering and being acquired for a pre-processing or a post-processing of the rendering of the 3D scene use;
using second patches for a pre-processing and/or a post-processing of the rendering of the 3D scene; and
rendering a viewport image from a point of view within the 3D scene by using first patches.

19. The device of claim 18 , wherein the second MVD content has a resolution lower than a resolution of the first MVD content.

20. The device of claim 18 , wherein the first atlas and the second atlas are a unique atlas, the unique atlas being associated with metadata indicating, for each patch whether the patch is the first patch or the second patch.

21. The device of claim 18 , wherein the second atlas associated with metadata indicating that the second patches are not intended to be used for view rendering and comprising a list of types of pre-processing or a post-processing of the rendering of the 3D scene.

22. The device of claim 21 , wherein a type of pre-processing or a post-processing of the rendering of the 3D scene belongs to a group of types of pre-processing or a post-processing of the rendering of the 3D scene comprising collision detection, interactions, and scene relighting.

Assignees

• Interdigital Ce Patent Holdings Sas

Inventors

• Dore Renaud
• Chupeau Bertrand
• Thudor Franck
• Fleureau Julien

Classifications

• H04N21/816

  involving special video data, e.g 3D video · CPC title

• H04N21/2662

  Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities · CPC title

• H04N21/2353

  specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata · CPC title

• G06T15/20

  Perspective computation · CPC title

• G06T15/08

  Volume rendering · CPC title