Method and apparatus for downscaling depth data for view plus depth data compression

Patent metadata

Field	Value
Publication number	US-9736455-B2
Application number	US-201414497687-A
Country	US
Kind code	B2
Filing date	Sep 26, 2014
Priority date	Jun 30, 2014
Publication date	Aug 15, 2017
Grant date	Aug 15, 2017

Abstract

A method, apparatus, and computer program product are disclosed for compression of a 2D-plus-depth representation based on spatial downsampling of an initial depth map. By utilizing the color image accompanying the initial depth map, it is possible to infer structural information that refines and reconstructs the initial depth map out of a heavily subsampled version of the depth map. In the process, no indexing of the exact positions of the subsampled depth values is needed, which leads to very efficient compression. An example method causes segmentation of the color image into a set of super-pixel segments, and causes downsampling of the initial depth map based on the set of super-pixel segments. The method subsequently causes generation and storage of a compressed representation based on the segmented color image and the downsampled depth map. A corresponding apparatus and computer program product are also provided.

First claim

What is claimed is:

1. A method comprising:
receiving a two-dimensional (2D)-plus-depth representation, the 2D-plus-depth representation comprising a color image and an original depth map;
causing segmentation of the color image into a set of super-pixel segments that group data of similar color, pattern, or texture;
causing, by a processor, downsampling of the original depth map based on the set of super-pixel segments;
causing generation of a compressed representation based on the segmented color image and the downsampled depth map, wherein the compressed representation includes one or more segmentation parameters associated with the segmentation of the color image and one or more downsampling parameters associated with the downsampling of the original depth map, wherein the compressed representation includes a binary tree that comprises depth intensity values and compressed representations for different layers of the binary tree; and
causing storage of the compressed representation.

2. The method of claim 1 , wherein the downsampling of the original depth map comprises:
identifying portions of the original depth map that correspond to segments of the set of super-pixel segments;
for each identified portion of the original depth map,
identifying a depth intensity value, and
modifying the portion of the original depth map based on the identified depth intensity value,

wherein the one or more downsampling parameters include the identified depth intensity values.

3. The method of claim 2 , wherein, in an instance in which the original depth map comprises floating point data, identifying the depth intensity value comprises:
applying a thin-plate spline fitting function to the portion of the original depth map.

4. The method of claim 2 , further comprising:
causing reconstruction of depth data from the downsampled depth map using an enhancement filter; and
causing determination of filter parameters of the enhancement filter and depth intensity values that maximize a quality of the reconstruction,
wherein the identified depth intensity values comprise the determined depth intensity values, and wherein the one or more downsampling parameters further include the determined filter parameters.

5. The method of claim 1 , further comprising:
calculating a size of the compressed representation and a quality of the compressed representation;
in an instance in which the calculated size fails to satisfy a size threshold or the calculated quality fails to satisfy a quality threshold,
causing segmentation of a segment from the set of segments into sub-segments,
causing further downsampling of the downsampled depth map based on the set of sub-segments,
causing updating of the compressed representation based on the further downsampling, and
re-calculating the size of the compressed representation and the quality of the compressed representation.

6. The method of claim 1 , further comprising:
generating the binary tree to capture index information associated with the segmentation of the color image and the downsampled depth map.

7. The method of claim 6 , further comprising:
causing application of predictive and arithmetic coding to the downsampled depth map prior to generation of the compressed representation.

8. The method of claim 1 , further comprising:
causing embedding of the downsampled depth map in the compressed representation.

9. The method of 8 , further comprising:
causing quantization of the color image by removing a set of least significant bits associated with the color image,
wherein the downsampled depth map is embedded in the quantized color image of the compressed representation.

10. The method of claim 1 , wherein the segmentation parameters and the downsampling parameters included in the compressed representation are located in a header of the compressed representation.

11. The method of claim 1 , wherein the color image is segmented into the set of super-pixel segments via simple linear iterative clustering or super-pixel extraction via energy-driven sampling.

12. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:
receive a two-dimensional (2D)-plus-depth representation, the 2D-plus-depth representation comprising a color image and an original depth map;
segment the color image into a set of super-pixel segments that group data of similar color, pattern, or texture;
downsample the original depth map based on the set of super-pixel segments;
generate a compressed representation based on the segmented color image and the downsampled depth map, wherein the compressed representation includes one or more segmentation parameters associated with the segmentation of the color image and one or more downsampling parameters associated with the downsampling of the original depth map, wherein the compressed representation includes a binary tree that comprises depth intensity values and compressed representations for different layers of the binary tree; and
cause storage of the compressed representation.

13. The apparatus of claim 12 , wherein the downsampling the original depth map comprises:
identifying portions of the original depth map that correspond to segments of the set of super-pixel segments;
for each identified portion of the original depth map,
identifying a depth intensity value, and
modifying the portion of the original depth map based on the identified depth intensity value,

wherein the one or more downsampling parameters include the identified depth intensity values.

14. The apparatus of claim 13 , wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:
reconstruct depth data from the downsampled depth map using an enhancement filter; and
determine filter parameters of the enhancement filter and depth intensity values that maximize a quality of the reconstruction,
wherein the identified depth intensity values comprise the determined depth intensity values, and wherein the one or more downsampling parameters further include the determined filter parameters.

15. The apparatus of claim 12 , wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:
calculate a size of the compressed representation and a quality of the compressed representation;
in an instance in which the calculated size fails to satisfy a size threshold or the calculated quality fails to satisfy a quality threshold,
segment a segment from the set of segments into sub-segments,
further downsample the downsampled depth map based on the sub-segments,
update the compressed representation based on the further downsampling, and
re-calculate the size of the compressed representation and the quality of the compressed representation.

16. The apparatus of claim 12 , wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:
generate the binary tree to capture index information associated with the segmentation of the color image and the downsampled depth map; and
causing application of predictive and arithmetic coding to the downsampled depth map prior to generation of the compressed representation.

17. The apparatus of claim 12 , wherein the at least one memory and the computer program code are configured to, with the at least

Assignees

• Nokia Technologies Oy

Inventors

• Georgiev Mihail
• Gotchev Atanas

Classifications

• H04N19/00

  Methods or arrangements for coding, decoding, compressing or decompressing digital video signals · CPC title

• H04N19/147

  according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) · CPC title

• H04N13/161 Primary

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N19/46

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title