Volumetric video supporting light effects

Patent metadata

Field	Value
Publication number	US-12579729-B2
Application number	US-202218285608-A
Country	US
Kind code	B2
Filing date	Apr 4, 2022
Priority date	Apr 7, 2021
Publication date	Mar 17, 2026
Grant date	Mar 17, 2026

Abstract

Methods, devices and data streams are proposed to encode, transport and decode 3D volumetric videos. The embodiments encompass the signaling of non-Lambertian patches together with their light reflection properties, so as to enable a ray-tracing based rendering engine to synthesize visually realistic virtual views with respect to light effects. A retro-compatible solution enabling conventional rendering engines to render such described 3D scene but without the advanced light effects is also described.

First claim

The invention claimed is:

1 . A method comprising:
for parts of a three-dimensional (3D) scene, obtaining first color patches and reflectance patches; for parts out of the 3D scene reflected in at least one part of the 3D scene, obtaining second color patches; generating a color atlas by packing the second color patches and a subset of the first color patches; generating a reflectance atlas by packing a subset of the reflectance patches; for each reflectance patch packed in the reflectance atlas,
generating first information encoding parameters of a bidirectional reflectance distribution function model of a light reflection on the reflectance patch, and
generating second information indicating a list of color patches reflected into the reflectance patch; and
encoding the color atlas, the reflectance atlas and the first information and the second information in a data stream.

2 . The method of claim 1 , wherein:
the subset of the first color patches packed in the color atlas is empty; and the subset of the reflectance patches packed in the reflectance atlas comprises every reflectance patch.

3 . The method of claim 1 , wherein:
the subset of the first color patches packed in the color atlas corresponds to Lambertian parts of the 3D scene; and the subset of the reflectance patches packed in the reflectance atlas corresponds to non-diffuse reflecting parts of the 3D scene.

4 . The method of claim 1 , wherein:
the subset of the first color patches packed in the color atlas comprises every first color patch; and the subset of the reflectance patches packed in the reflectance atlas corresponds to non-diffuse reflecting parts of the 3D scene.

5 . The method of claim 1 , wherein the bidirectional reflectance distribution function model is a Phong model.

6 . The method of claim 1 , further comprising generating a surface normal atlas by packing surface normal patches corresponding to the subset of the reflectance patches in the reflectance atlas.

7 . The method of claim 1 , further comprising:
for parts of the 3D scene, obtaining first depth patches; for parts out of the 3D scene, obtaining second depth patches; generating a depth atlas by packing first and second depth patches; and encoding the depth atlas in the data stream.

8 . A device comprising:
a processor configured to:
for parts of a three-dimensional (3D) scene, obtain first color patches and reflectance patches;
for parts out of the 3D scene reflected in at least one part of the 3D scene, obtain second color patches;
generate a color atlas by packing the second color patches and a subset of the first color patches;
generate a reflectance atlas by packing a subset of the reflectance patches;
for each reflectance patch packed in the reflectance atlas,
generate first information encoding parameters of a bidirectional reflectance distribution function model of a light reflection on the reflectance patch, and
generate second information indicating a list of color patches reflected into the reflectance patch; and

encode the color atlas, the reflectance atlas, the first information and the second information in a data stream.

9 . The device of claim 8 , wherein:
the subset of the first color patches packed in the color atlas is empty; and the subset of the reflectance patches packed in the reflectance atlas comprises every reflectance patch.

10 . The device of claim 8 , wherein:
the subset of the first color patches packed in the color atlas corresponds to Lambertian parts of the 3D scene; and the subset of the reflectance patches packed in the reflectance atlas corresponds to non-diffuse reflecting parts of the 3D scene.

11 . The device of claim 8 , wherein:
the subset of the first color patches packed in the color atlas comprises every first color patch; and the subset of the reflectance patches packed in the reflectance atlas corresponds to non-diffuse reflecting parts of the 3D scene.

12 . The device of claim 8 , wherein the bidirectional reflectance distribution function model is a Phong model.

13 . The device of claim 8 , wherein the processor is further configured to generate a surface normal atlas by packing surface normal patches corresponding to the subset of the reflectance patches in the reflectance atlas.

14 . The device of claim 8 , wherein the processor is further configured to:
for parts of the 3D scene, obtain first depth patches; for parts out of the 3D scene, obtain second depth patches; generate a depth atlas by packing first and second depth patches; and encode the depth atlas in the data stream.

15 . A method of rendering a 3D scene, the method comprising:
decoding from a data stream:
a color atlas packing first color patches corresponding to parts of the 3D scene and second color patches corresponding to parts out of the 3D scene reflected in at least one part of the 3D scene;
a reflectance atlas packing reflectance patches corresponding to parts of the 3D scene;
information signaling a rendering mode determined according to the first color patches and the reflectance patches; and
for each reflectance patch packed in the reflectance atlas,
first information encoding parameters of a bidirectional reflectance distribution function model of a light reflection on the reflectance patch, and
second information indicating a list of color patches reflected into the reflectance patch;

and rendering the 3D scene by un-projecting the first and second color patches using ray tracing for reflectance patches according to first and second information and associated color patches.

16 . The method of claim 15 , wherein the bidirectional reflectance distribution function model is a Phong model.

17 . The method of claim 15 , further comprising decoding, from the data stream, a surface normal atlas packing surface normal patches corresponding to a subset of the reflectance patches in the reflectance atlas and using surface normal patches for ray tracing.

18 . The method of claim 15 , further comprising:
decoding, from the data stream, a depth atlas packing first depth patches corresponding to parts of a 3D scene and second depth patches corresponding to parts out of the 3D scene reflected in at least one part of the 3D scene; and rendering the 3D scene by un-projecting the first and second color patches according to the first and second depth patches.

19 . A device comprising:
a processor configured to:
decode from a data stream:
a color atlas packing first color patches corresponding to parts of a three-dimensional (3D) scene and second color patches corresponding to parts out of the 3D scene reflected in at least one part of the 3D scene;
a reflectance atlas packing reflectance patches corresponding to parts of the 3D scene;
information signaling a rendering mode determined according to the first color patches and the reflectance patches; and
for each reflectance patch packed in the reflectance atlas,
first information encoding parameters of a bidirectional reflectance distribution function model of a light reflection on the reflectance patch, and
second information indicating a list of color patches reflected into the reflectance patch;

and
render the 3D scene by un-projecting the first and second color patches by using ray tracing for reflectance patches according to first and second information and associated color patches.

20 . The device of claim 19 , wherein the bidirectional reflectance distribution function model i

Assignees

• Interdigital Ce Patent Holdings Sas

Inventors

• Chupeau Bertrand
• Dore Renaud
• Fleureau Julien
• Thudor Franck

Classifications

• G06V10/56

  relating to colour · CPC title

• H04N19/597 Primary

  specially adapted for multi-view video sequence encoding · CPC title

• H04N13/351

  for displaying simultaneously · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title