Multi view synthesis method and display devices with spatial and inter-view consistency

Patent metadata

Field	Value
Publication number	US-10148930-B2
Application number	US-201615189876-A
Country	US
Kind code	B2
Filing date	Jun 22, 2016
Priority date	Oct 26, 2011
Publication date	Dec 4, 2018
Grant date	Dec 4, 2018

Abstract

An embodiment of the invention is a method that provides common data and enforces multiple view consistency. In a preferred embodiment, a coordinate alignment matrix is generated after initial warped views are formed. The matrix is used efficiently to repair holes/occlusions when synthesizing final views. In another preferred embodiment, a common background layer is generated based upon initial warped views. Holes are repaired on the common background layer, which can then be used for synthesizing final views. The cost of the latter method is sub-linear, meaning that additional views can be generated with lesser cost. Preferred methods of the invention also use a placement matrix to generate initial warped views, which simplifies warping to a look-up operation.

First claim

The invention claimed is:

1. A method for rendering multiple view data from a stereoscopic video data stream, comprising:
receiving or estimating a disparity map from the stereoscopic data;
forming a placement matrix that provides pixel locations in an intended virtual view for pixels in the multiple view data,
wherein the placement matrix at position α is defined as:
P k ( x,y +( k ) D k ( x,y ))= y,   (1)

where x and y are the row and column indices respectively, where D is the disparity map, k is a reference image position indicating a view from the multiple view data, and the placement matrix P k (x, y)=ŷ provides the pixel location (x′, ŷ) in the reference image that should be mapped on the pixel location (x′, y′) in the virtual view after warping;
warping pixels onto multiple views via a look-up from the placement matrix;
preparing hole repair data that is common for the multiple views;
generating a common background layer by combining the pixels belonging to background regions onto a single layer and filling holes directly on the common background layer using the hole repair data;
generating multiple final views while conducting hole repair with the data that is common.

2. The method of claim 1 , further comprising, after said forming, detecting defects in the placement matrix and correcting the defects by selecting and using surrounding information from background or foreground objects associated with the defect.

3. The method of claim 2 , further comprising refining locations having multiple candidate pixels to remove ghost images.

4. The method of claim 1 , wherein said warping pixels comprises using a candidate pixel for a certain location when it is the sole candidate for that location and when there are multiple candidates for a certain location from both views, the candidate that is closer to the camera is used initially, while locations having two candidate pixels are remembered for use in hole repair.

5. The method of claim 1 , wherein the data that is common comprises a coordinate alignment matrix that identifies occluded regions on a common reference image, and holes are repaired individually from a local background layer for generating the final views.

6. The method of claim 1 , wherein the stereoscopic data consists of two views.

7. The method of claim 1 , wherein the stereoscopic data comprises three or more views.

8. The method of claim 1 , further comprising a preliminary step of refining the disparity map to eliminate pixels that fall outside of a predetermined range from a mode and filling such pixels in the disparity map based upon neighboring pixels in the disparity map.

9. The method of claim 1 , wherein the placement matrix exhibits a non-decreasing trend from left to right, and includes jumps around object boundaries.

10. The method of claim 1 , wherein said warping comprises warping operation for both left and the right view followed by blending warped disparity maps and images into a single disparity map.

11. The method of claim 1 , wherein said generating comprises extracting local background pixels that neighbor occluded regions in the multiple views after said directly from reference images used by said warping.

12. The method of claim 1 , wherein said preparing comprises patch matching and exemplar based inpainting.

13. A method for rendering multiple view data from a stereoscopic video data stream, comprising:
generating a placement matrix for a predetermined number of views and warping the views with the placement matrix using a look-up operation,
wherein the placement matrix at position α is defined as:
P k ( x,y +( k ) D k ( x,y ))= y,   (1)

where x and y are the row and column indices respectively, where D is a disparity map, k is a reference image position indicating a view from the multiple view data, and the placement matrix P k (x, y)=ŷ provides the pixel location (x′, ŷ) in the reference image that should be mapped on the pixel location (x′, y′) in the virtual view after warping;
conducting hole repair for the views, wherein said conducting hole repair comprises generating a common background layer for warped views generated with the placement matrix by combining the pixels belonging to background regions onto a single layer, repairing holes on the common background layer and repairing holes in warped views individually with respect to the common background layer; and
providing display data to a 2D display device with a lenticular screen for the predetermined number of views for producing a 3D display.

Assignees

• Univ California

Inventors

• Nguyen Truong Q
• Tran Lam
• Bal Can
• Khoshabeh Ramsin

Classifications

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• G06T15/08

  Volume rendering · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• H04N13/305

  using lenticular lenses, e.g. arrangements of cylindrical lenses · CPC title