Representation and coding of multi-view images using tapestry encoding

Patent metadata

Field	Value
Publication number	US-9451232-B2
Application number	US-201214237548-A
Country	US
Kind code	B2
Filing date	Sep 27, 2012
Priority date	Sep 29, 2011
Publication date	Sep 20, 2016
Grant date	Sep 20, 2016

Abstract

Representation and coding of multi-view images using tapestry encoding are described. A tapestry comprises information on a tapestry image and a world coordinates map associated with the tapestry image, each of which may contain information from one or more perspectives of a scene. Perspective images of a scene can be generated from the tapestry image and the world coordinates map.

First claim

What is claimed is:

1. A method for generating at least one view of a scene from a tapestry image, each of the at least one view being associated with one desired viewpoint of the scene, the method comprising:
providing the tapestry image; wherein the tapestry image comprises a 2 D array of pixels comprising information from a plurality of views associated with the scene;
providing a coordinates map associated with the tapestry image; wherein position data associated with the pixels of the tapestry image comprise depth data for each pixel in the tapestry image; and wherein the position data associated with the pixels of the tapestry image further comprise horizontal disparity data and/or vertical disparity data for each pixel in the tapestry image;
deriving one or more views of the scene based on the tapestry image and the coordinates map.

2. The method according to claim 1 , further comprising:
providing reference images of the scene, each reference image being associated with one desired viewpoint; and
generating one or more correction layers, each correction layer being associated with one desired viewpoint, wherein a correction layer associated with a particular desired viewpoint is generated based on the reference image associated with the particular desired viewpoint and the derived view associated with the same particular desired viewpoint,
wherein the generating of the at least one view is further based on the one or more correction layers.

3. The method according to claim 2 , wherein each of the one or more correction layers is based on a difference between the reference image associated with the particular desired viewpoint and the derived view associated with the same particular desired viewpoint.

4. The method according to claim 3 , wherein the difference comprises subtracting each pixel in the reference image associated with the particular desired viewpoint from a corresponding pixel in the derived view associated with the same particular desired viewpoint.

5. The method according to claim 2 , wherein the deriving comprises adding, for each view in the at least one view, a view from the at least one view with a correction layer from the one or more correction layers, and wherein the view and the correction layer are associated with the same desired viewpoint.

6. The method according to claim 1 , wherein the deriving comprises performing textured area filling based on the tapestry image and the coordinates map.

7. The method according to claim 1 , wherein the providing the tapestry image comprises:
taking pictures of the scene from a plurality of viewpoints of the scene; and
processing the taken pictures to generate the tapestry image.

8. The method according to claim 1 , wherein the providing the tapestry image comprises:
generating pictures, using a computer, of the scene for a plurality of viewpoints of the scene; and
processing the computer generated pictures to generate the tapestry image.

9. The method according to claim 6 , wherein the processing the tapestry image comprises:
selecting views from the plurality of views of the scene; and
performing a combination based on the selected views to generate the tapestry image.

10. The method according to claim 1 , wherein the providing the tapestry image comprises:
performing an analysis of the scene to obtain a plurality of features associated with the scene;
prioritizing each feature in the plurality of features from most important to least important; and
generating the tapestry image such that higher resolution is allocated to features of higher importance than to features of lower importance.

11. The method according claim 10 , wherein the performing an analysis is based on one or more of manual allocation of importance, frequency analysis, and saliency analysis.

12. The method according to claim 1 , further comprising generating a camera grid, wherein the camera grid comprises identification information of each image in the at least one image of a scene from a tapestry image.

13. The method according to claim 12 , wherein the identification information of a particular image comprises at least one of view position, view angle, view direction, orientation, and horizontal and vertical size of the particular image.

14. An encoding system adapted to receive a plurality of input views of a scene, comprising:
a tapestry generator adapted to generate a tapestry image and a coordinates map based on the plurality of input views; wherein the tapestry image comprises a 2 D array of pixels comprising information from the plurality of input views of the scene; wherein position data associated with the pixels of the tapestry image comprise depth data for each pixel in the tapestry image; and wherein the position data associated with the pixels of the tapestry image further comprise horizontal disparity data and/or vertical disparity data for each pixel in the tapestry image; and
an encoder adapted to encode the tapestry image and the coordinates map into a bitstream.

15. The encoding system according to claim 14 , further comprising:
a view generation unit adapted to derive one or more views of the scene based on the tapestry image and the coordinates map; and
a subtractor coupled with the view generation unit and adapted to generate a difference between the one or more derived views and the one or more input views, each difference being between a derived view of a particular viewpoint and an input view of the same particular viewpoint, thus generating one or more correction layers,
wherein the encoder is adapted to encode the tapestry image, the coordinates map, and the one or more correction layers into a bitstream.

16. The encoding system according to claim 14 , wherein:
the encoding system is further adapted to receive a camera grid, each point in the camera grid being associated with one input view of the scene, and
the view generation unit is adapted to derive the one or more views of the scene based further on the camera grid.

17. The encoding system according to claim 16 , wherein each point in the camera grid comprises, for each input view of the scene, information on one or more of view position, view angle, view direction, orientation, and horizontal and vertical size associated with the input view.

18. The encoding system according to claim 16 , wherein each point in the camera grid comprises, for each input view of the scene, information on one or more of view position, view angle, view direction, orientation, horizontal and vertical size, and correction layer associated with the input view.

19. A decoding system adapted to receive a bitstream associated with a scene and adapted to generate one or more output images, comprising:
a decoder adapted to decode a tapestry image and a coordinates map from the bitstream, wherein the tapestry image and the coordinates map are associated with a plurality of input views of the scene; wherein the tapestry image comprises a 2 D array of pixels comprising information from the plurality of input views of the scene; wherein position data associated with the pixels of the tapestry image comprise depth data for each pixel in the tapestry image; and wherein the position data associated with the pixels of the tapestry image further comprise horizontal disparity data and/or vertical disparity data for each pixel in the tapestry image; and
a view generation unit adapted to derive the one or more output images of the scene based on the tapestry image and the coordinates map.

20. The decoding syst

Assignees

• Dolby Laboratories Licensing Corp

Inventors

• Ward Gregory John
• Daly Scott

Classifications

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title

• H04N13/161

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/139

  Format conversion, e.g. of frame-rate or size · CPC title

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• H04N13/0048

  Electricity · mapped topic