Volumetric video with long-term patch entity memory

Patent metadata

Field	Value
Publication number	US-12430845-B2
Application number	US-202218285947-A
Country	US
Kind code	B2
Filing date	Apr 12, 2022
Priority date	Apr 15, 2021
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

Methods and devices for encoding and decoding 3D scenes of an immersive video are disclosed with the related data stream. Taking advantage of the fact that parts of the 3D scene does not change from one frame of the video to another, even when the cameras move and/or the lighting condition change, dynamic and static parts of the 3D scene are encoded in different atlases. In an embodiment, static patch atlases are repeated between two updates. In another embodiment, only updates are encoded and transmitted. The decoder maintains a memory associating entity of the 3D scene with a set of static patches that is updated with instructions linked to update static patch atlases. At each presentation time the static part of the 3D scene to decode is obtained from the memory.

First claim

The invention claimed is:

1. A method comprising:
obtaining, from a data stream, a second atlas packing one or more static patches for a first three-dimensional (3D) scene of a sequence of 3D scenes, a static patch being associated with an entity of the first 3D scene, an entity being a part of the first 3D scene;
updating one or more entries of a local memory with the one or more static patches;
obtaining, from the data stream, a first atlas packing one or more-dynamic patches for a second 3D scene of the sequence of 3D scenes;
obtaining entries from the local memory corresponding to static patches associated with an entity visible in a viewport to render; and
rendering the second 3D scene by inverse projecting the one or more dynamic patches and static patches of obtained entries.

2. The method of claim 1 , further comprising:
obtaining, from the data stream, metadata indicating, for an entity of the first 3D scene, a set of instructions, an instruction comprising:
creating, for the first 3D scene, a new entry in the local memory associating the entity with a subset of patches of the second atlas;
updating, for the first 3D scene, the entry of the local memory associating the entity with second patches, with a subset of patches of the second atlas; and
deleting from the local memory, the entry associating the entity with second patches; and

updating the local memory by applying the instruction indicated in the metadata.

3. The method of claim 1 , wherein an entry associates an entity of the first 3D scene with static patches of the first 3D scene and with one or more adaptation parameters representative of a camera movement and/or of a lighting change; and the method further comprising adapting the one or more static patches according to the one or more adaptation parameters.

4. The method of claim 3 , wherein the one or more adaptation parameters are representative of one or more parameters of a parametrized function, the parametrized function belonging to a set of parametrized functions comprising:
polynomial functions;
Bezier functions; and
Spline functions.

5. The method of one of claim 1 , wherein the entries from the local memory for the first 3D scene are the entries from the local memory for a previous 3D scene in the sequence of 3D scenes.

6. The method of claim 1 , wherein the data stream comprises a first sub-bitstream carrying first atlases and a second sub-bitstream carrying second atlases.

7. A method comprising:
obtaining a first atlas packing one or more dynamic patches of a three-dimensional (3D) scene of a sequence of 3D scenes;
obtaining one or more static patches of the 3D scene, a static patch being associated with an entity, an entity being a part of the 3D scene;
encoding the first atlas in a data stream;
on condition that at least one static patch differs from static patches of a previous 3D scene in the sequence of 3D scenes:
packing one or more static patches in a second atlas and generating metadata indicating, for an entity associated with a static patch, an instruction for a client to:
create, for the 3D scene, a new entry in a local memory associating the entity with static patches associated with the entity; or
update, for the 3D scene, the entry of the local memory associating the entity with static patches, with static patches associated with the entity; or
delete from the local memory, the entry associating the entity with static patches; and

encoding the second atlas and the metadata in the data stream.

8. The method of claim 7 , wherein an entry associates an entity of the 3D scene with static patches for the 3D scene and one or more adaptation parameters representative of a camera movement and/or of a lighting change; and the method further comprising encoding the one or more adaptation parameters in the metadata.

9. The method of claim 8 , wherein the adaptation parameters are representative of one or more parameters of a parametrized function, the parametrized function belonging to a set of parametrized functions comprising:
polynomial functions;
Bezier functions; and
Spline functions.

10. The method of claim 7 , wherein the first atlas is encoded in a first sub-bitstream of the data stream and the second atlas is encoded in a second sub-bitstream of the data stream.

11. A device comprising a processor configured to:
obtain, from a data stream, a second atlas packing one or more static patches for a first 3D scene of a sequence of 3D scenes, a static patch being associated with an entity of the 3D scene, an entity being a part of the first 3D scene ;
update one or more entries of a local memory with the one or more static patches;
obtain, from the data stream, a first atlas packing one or more dynamic patches for a second 3D scene of the sequence of 3D scenes;
obtain entries from a local memory corresponding to static patches associated with entities visible in a viewport to render; and
render the second 3D scene by inverse projecting one or more dynamic patches and static patches of obtained entries.

12. The device of claim 11 , wherein the processor is further configured to:
obtain, from the data stream, metadata indicating, for an entity of the first 3D scene, a set of instructions, an instruction comprising:
creating, for the first 3D scene, a new entry in the local memory associating the entity with a subset of patches of the second atlas;
updating, for the first 3D scene, the entry of the local memory associating the entity with second patches, with a subset of patches of the second atlas; and
deleting from the local memory, the entry associating the entity with second patches; and

update the local memory by applying the instruction indicated in the metadata.

13. The device of claim 11 , wherein an entry associates an entity of the first 3D scene with static patches for the first 3D scene and with one or more adaptation parameters representative of a camera movement and/or of a lighting change; and the processor being further configured to adapt the static patches according to the one or more adaptation parameters.

14. The device of claim 13 , wherein the one or more adaptation parameters are representative of parameters of a parametrized function, the parametrized function belonging to a set of parametrized functions comprising:
polynomial functions;
Bezier functions; and
Spline functions.

15. The device of claim 11 , wherein the entries from the local memory for the first 3D scene are the entries of the local memory for a previous 3D scene in the sequence of 3D scenes.

16. The device of claim 11 , wherein the data stream comprises a first sub-bitstream carrying first atlases and a second sub-bitstream carrying second atlases.

17. A device comprising a processor configured to:
obtain a first atlas packing one or more dynamic patches of a 3D scene of a sequence of 3D scenes;
obtain one or more static patches of the 3D scene, a static patch being associated with an entity, an entity being a part of the 3D scene;
encode the first atlas in a data stream;
on condition that at least one static patch differs from the static patches of a previous 3D scene in the sequence of 3D scenes:
pack static patches in a second atlas and generating metadata indicating, for an entity associated with a static patch, an instruction for a client to:
create, for the 3D scene, a new entry in a local memory associating the entity with static patches associated with the entity; or
update, for the 3D scene, the entry of the local memory associating the entity with static patches, with stati

Assignees

• Interdigital Ce Patent Holdings Sas

Inventors

• Chupeau Bertrand
• Dore Renaud
• Briand Gerard
• Fleureau Julien
• Gendrot Remy
• Thudor Franck
• Sabater Neus
• Tapie Thierry

Classifications

• H04N13/279

  the virtual viewpoint locations being selected by the viewers or determined by tracking · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• G06T15/10

  Geometric effects · CPC title

• H04N13/332

  Displays for viewing with the aid of special glasses or head-mounted displays [HMD] · CPC title

• H04N13/349

  Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking (for viewing without the aid of special glasses using fly-eye lenses H04N13/307) · CPC title