Volumetric video with auxiliary patches

Patent metadata

Field	Value
Publication number	US-2025024076-A1
Application number	US-202418804382-A
Country	US
Kind code	A1
Filing date	Aug 14, 2024
Priority date	Dec 19, 2019
Publication date	Jan 16, 2025
Grant date	—

Abstract

Methods and devices for encoding and decoding data representative of a 3D scene are disclosed. A set of first patches is generated from a first MVD content acquired from a first region of the 3D scene. A patch is a part of one of the views of the MVD content. A set of second patches is generated from a second MVD content acquired from a second region of the 3D scene. An atlas packing first and second patches is generated and associated with metadata indicating, for a patch of the atlas, whether the patch is a first or a second patch At the decoding side, first patches are used for rendering the viewport image and second patches are used for pre-processing or post-processing the viewport image.

First claim

1 . A method for encoding a 3D scene in a data stream, the method comprising:
generating a set of first patches from a first multi-view-plus-depth (MVD) content acquired for a rendering of the 3D scene, a first patch being a part of one of the views of the first MVD content; generating a set of second patches from a second MVD content, wherein the second patches are to be used for at least one of collision detection, interactions, and scene relighting, a second patch being a part of one of the views of the second MVD content; generating a first atlas packing said first patches and a second atlas packing said second patches, an atlas being an image packing patches according to an atlas layout; and encoding said first and second atlases in said data stream.

2 . The method of claim 1 , wherein the second MVD content is acquired in a resolution lower than a resolution of the first MVD content.

3 . The method of claim 1 , wherein a patch is a part of one view of a MVD content obtained by removing information redundancy between views of the MVD content.

4 . The method of claim 1 , wherein said first and second patches are packed in a common atlas, the common atlas being associated with metadata indicating, for each patch, whether the patch is the first patch or the second patch.

5 . The method of claim 1 , wherein the second atlas associated with metadata indicating that the second patches are to be used for at least one of collision detection, interactions, and scene relighting.

6 . A non-transitory computer readable storage device storing instructions which, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .

7 . A device for encoding a 3D scene in a data stream, the device comprising a memory associated with a processor configured for:
generating a set of first patches from a first multi-view-plus-depth (MVD) content acquired for a rendering of the 3D scene, a first patch being a part of one of the views of the first MVD content; generating a set of second patches from a second MVD content, wherein the second patches are to be used for at least one of collision detection, interactions, and scene relighting, a second patch being a part of one of the views of the second MVD content; generating a first atlas packing said first patches and a second atlas packing said second patches, an atlas being an image packing patches according to an atlas layout; and encoding said first and second atlases in said data stream.

8 . The device of claim 7 , wherein the second MVD content is acquired in a resolution lower than a resolution of the first MVD content.

9 . The device of claim 7 , wherein a patch is a part of one view of a MVD content obtained by removing information redundancy between views of the MVD content.

10 . The device of claim 7 , wherein said first and second patches are packed in a unique atlas, the unique atlas being associated with metadata indicating, for each patch, whether the patch is the first patch or the second patch.

11 . The device of claim 7 , wherein the second atlas associated with metadata indicating that the second patches are to be used for at least one of collision detection, interactions, and scene relighting.

12 . A method for decoding a 3D scene from a data stream, the method comprising:
decoding said data stream to retrieve a first atlas and a second atlas, wherein the first atlas and the second atlas are images packing first or second patches respectively according to an atlas layout, a first patch being a part of one view of a first MVD content acquired for a rendering of the 3D scene, a second patch being a part of one view of a second MVD content, and being acquired for at least one of collision detection, interactions, and scene relighting; using second patches for at least one of collision detection, interactions, and scene relighting when rendering the 3D scene; and rendering a viewport image from a point of view within the 3D scene by using first patches.

13 . The method of claim 12 , wherein the second MVD content has a resolution lower than a resolution of the first MVD content.

14 . The method of claim 12 , wherein the first atlas and the second atlas are a unique atlas, the unique atlas being associated with metadata indicating, for each patch, whether the patch is the first patch or the second patch.

15 . The method of claim 12 , wherein the second atlas associated with metadata indicating that the second patches are to be used for at least one of collision detection, interactions, and scene relighting.

16 . A non-transitory computer readable storage device storing instructions which, when executed by one or more processors, cause the one or more processors to perform the method of claim 12 .

17 . A device for decoding a 3D scene from a data stream, the device comprising a processor configured for:
decoding said data stream to retrieve a first atlas and a second atlas, wherein the first atlas and the second atlas are images packing first or second patches respectively according to an atlas layout, a first patch being a part of one view of a first MVD content acquired for a rendering of the 3D scene, a second patch being a part of one view of a second MVD content, being acquired for at least one of collision detection, interactions, and scene relighting; using second patches for at least one of collision detection, interactions, and scene relighting when rendering the 3D scene; and rendering a viewport image from a point of view within the 3D scene by using first patches.

18 . The device of claim 17 , wherein the second MVD content has a resolution lower than a resolution of the first MVD content.

19 . The device of claim 17 , wherein the first atlas and the second atlas are a unique atlas, the unique atlas being associated with metadata indicating, for each patch, whether the patch is the first patch or the second patch.

20 . The device of claim 17 , wherein the second atlas associated with metadata indicating that the second patches are to be used for at least one of collision detection, interactions, and scene relighting.

Assignees

• Interdigital Ce Patent Holdings Sas

Inventors

• Dore Renaud
• Chupeau Bertrand
• Thudor Franck
• Fleureau Julien

Classifications

• H04N21/816

  involving special video data, e.g 3D video · CPC title

• H04N21/2662

  Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities · CPC title

• H04N21/2353

  specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata · CPC title

• G06T15/20

  Perspective computation · CPC title

• G06T15/08

  Volume rendering · CPC title