Method of encoding a video data signal for use with a multi-view rendering device

Patent metadata

Field	Value
Publication number	US-9826212-B2
Application number	US-201414889902-A
Country	US
Kind code	B2
Filing date	Apr 30, 2014
Priority date	May 10, 2013
Publication date	Nov 21, 2017
Grant date	Nov 21, 2017

Abstract

The present invention relates to a method of encoding a video data signal for use with a multi-view rendering device, a method of decoding the video data signal, the video data signal, an encoder of the video data signal, a decoder of the video data signal, a computer program product comprising instructions for encoding the video data signal and a computer program product comprising instructions for decoding a video data signal. The method of encoding provides ( 401 ) a first image ( 10 ) of a scene associated with a first viewpoint, a depth map ( 20 ) associated with the first image, metadata ( 30 ) for use in depth map processing or rendering one or more views for further viewpoints by the multi-view rendering device, and generates ( 404 ) the video data signal. The video data signal comprises video frames partitioned in sub-images comprising a sub-image based on the first image and a depth sub-image based on the depth map, and metadata encoded in a color component of the depth sub-image.

First claim

The invention claimed is:

1. A method of encoding a video data signal for use in a multi-view rendering device, the method comprising:
providing a first image of a scene, wherein the scene is associated with a first viewpoint;
providing a depth map associated with the first image;
providing metadata, wherein the metadata is arranged for
depth map processing or rendering one or more views for further viewpoints;

generating the video data signal, the video data signal comprising video frames partitioned into sub-images, the sub-images comprising:
a first sub-image based on the first image;
a first depth sub-image based on the depth map; and
wherein the metadata is encoded in a color component of the first depth sub-image, the metadata being repeatedly inserted in subsequent video frames, whether or not there was a change in the metadata, and the metadata comprising change information indicating whether the metadata has changed since the previous frame so as to enable said use of the metadata frame accurate or fixed for several frames.

2. The method of encoding according to claim 1 , wherein
the video data signal is arranged in luminance and chrominance values, and the depth map is encoded in the luminance values of the first depth sub-image and the metadata is encoded in the chrominance values of the first depth sub-image; or
the video data signal is arranged in R, G and B components, and the depth map is encoded in a first component of the R, G and B components of the first depth sub-image and the metadata is encoded a further component of the R, G and B components of the first depth sub-image.

3. The method of encoding according to claim 1 , wherein the video frame comprised in the video data signal comprises an uncompressed two-dimensional array of data.

4. The method of encoding according to claim 1 , the method further comprising
transmitting the video data signal over an interface using one or more bit-serial data lines to the multi-view rendering device.

5. The method of encoding according to claim 1 , wherein the metadata is inserted in the video data signal
for every field, or
for every frame, or
for every group of pictures, and/or
for every scene.

6. The method of encoding according to claim 1 , wherein the metadata is provided with a checksum.

7. The method of encoding according to claim 1 the method further comprising:
providing a second image of a scene associated with a second viewpoint;
providing a second depth map associated with the second image,
wherein generating the video data signal, further comprises:
a second sub-image based on the second image,
a second depth sub-image based on the second depth map, and the video data signal.

8. A video data signal, the video data signal for use in a multi-view rendering device, the video data signal comprising:
a first image of a scene, wherein the scene is associated with a first viewpoint,
a depth map associated with the first image;
a metadata, wherein the metadata is arranged for
depth map processing or
rendering one or more views for further view points;

wherein the video data signal comprises video frames partitioned into sub-images, the sub-images comprising:
a first sub-image based on the first image;
a first depth sub-images based on the depth map; and
wherein the metadata is encoded in a color component of the first depth sub-image, the metadata being repeatedly inserted in subsequent video frames, whether or not there was a change in the metadata, and the metadata comprising change information indicating whether the metadata has changed since the previous frame so as to enable said use of the metadata frame accurate or fixed for several frames.

9. A data carrier, comprising the video data signal of claim 8 in a non-transitory form.

10. A method of decoding a video data signal, the method comprising:
receiving the video data signal, the video data signal comprising:
a first image of a scene, wherein the scene is associated with a first viewpoint;
a depth map associated with the first image;
metadata, wherein the metadata is arranged for
depth map processing or
rendering multiple views;

video frames partitioned in sub-images, the sub-images comprising:
a first sub-image based on the first image;
a first depth sub-image based on the depth map; and
wherein the metadata is encoded in the first depth sub-image,
de-multiplexing so as to obtain access to the individual components and to retrieve the metadata from a color component of the first depth sub-image, the metadata being repeatedly inserted in subsequent video frames, whether or not there was a change in the metadata, and the metadata comprising change information indicating whether the metadata has changed since the previous frame; and
performing at least one of the following steps:
depth map processing of the depth map in dependence of the metadata;
rendering one or more views for further viewpoints in dependence of the metadata,

wherein the method comprises a control process for handling the metadata processing and for using the change information in order to determine whether to process the metadata further in dependence of the change information either frame accurate or fixed for several frames.

11. A decoder for decoding a video data signal, the decoder comprising:
a receiver, arranged to receive the video data signal, the video data signal comprising:
a first image of a scene associated with a first viewpoint;
a depth map associated with the first image; and
metadata for use in
depth map processing or
rendering multiple views,

wherein the video data signal comprises video frames partitioned in sub-images comprising:
a first sub-image based on the first image;
a first depth sub-image based on the depth map; and
the metadata encoded in the first depth sub-image,

a de-multiplexer, the de-multiplexer arranged to de-multiplex the video data signal so as to obtain access to the individual components and to retrieve the metadata from a color component of the first depth sub-image, the metadata being repeatedly inserted in subsequent video frames, whether or not there was a change in the metadata, and the metadata comprising information indicating whether the metadata has changed since the previous frame; and
at least one of:
a depth map processor arranged to process the depth map in dependence of the metadata; or
a rendering unit, the rendering unit arranged to rendering one or more views in dependence of the metadata,

wherein the decoder comprises a control processor for handling the metadata processing and for using the change information in order to determine whether to process the metadata further in dependence of the change information either frame accurate or fixed for several frames.

12. The decoder according to claim 11 , the decoder further comprising a multi-view stereoscopic display unit, for displaying the rendered one or more views.

13. The decoder according to claim 11 further arranged to decode any one of the video data signals.

14. A computer program product comprising instructions for causing a processor system to perform the method according to claim 1 .

15. An encoder of a video data signal for use in a multi-view rendering device, the encoder comprising:
a acquisition unit arranged to acquire a first image of a scene, wherein the scene is associated with a first viewpoint;
a acquisition unit arranged to acquire a depth map associated with the first image;
a acquisition unit arranged to acquire metadata for use in
depth map processing or
rendering one or m

Assignees

• Koninklijke Philips Nv

Inventors

• Newton Philip Steven
• De Haan Wiebe

Classifications

• H04N19/186

  the unit being a colour or a chrominance component · CPC title

• H04N19/46 Primary

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title

• H04N19/179

  the unit being a scene or a shot · CPC title

• H04N21/21805

  enabling multiple viewpoints, e.g. using a plurality of cameras · CPC title

• H04N19/17

  the unit being an image region, e.g. an object · CPC title