Synthesizing views based on image domain warping

Patent metadata

Field	Value
Publication number	US-9445072-B2
Application number	US-201213601363-A
Country	US
Kind code	B2
Filing date	Aug 31, 2012
Priority date	Nov 11, 2009
Publication date	Sep 13, 2016
Grant date	Sep 13, 2016

Abstract

Techniques are disclosed for generating autostereoscopic video content. A multiscopic video frame is received that includes a first image and a second image. The first and second images are analyzed to determine a set of image characteristics. A mapping function is determined based on the set of image characteristics. At least a third image is generated based on the mapping function and added to the multiscopic video frame.

First claim

What is claimed is:

1. A computer-implemented method of multiscopic video augmentation based on non-linear warp functions, the computer-implemented method comprising:
receiving a multiscopic video frame comprising at least a first image and a second image, at least one image of which has a first set of pixels and a second set of pixels;
analyzing the first image and the second image of the multiscopic video frame in order to determine a set of image characteristics including a set of sparse disparities between the first and second images;
determining, based on the set of image characteristics, a mapping function comprising a non-linear warp function configured to, in an image domain, warp the first set of pixels to a greater extent than the second set of pixels; and
generating, by operation of one or more computer processors, at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

2. The computer-implemented method of claim 1 , wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

3. The computer-implemented method of claim 2 , wherein the saliency map is determined by analyzing a predefined set of image attributes based on a quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation.

4. The computer-implemented method of claim 3 , wherein the set of image characteristics further includes an edge map of at least one of the first image and the second image, wherein the non-linear warp function is determined by optimizing an energy equation based on the set of image characteristics and a set of associated weights.

5. The computer-implemented method of claim 4 , wherein the multiscopic video frame comprises a stereoscopic video frame, wherein the third image is generated further based on at least one of interpolation and extrapolation;
wherein the set of sparse disparities between the first image and second image are determined based on feature matching and optical flow analysis;
wherein the edge map is determined based on edge detection and vertical line detection, wherein the edge detection is based on a Canny algorithm, wherein the vertical line detection is based on a Hough transform;
wherein the determined mapping function is configured to, when generating the third image, preserve vertical edges and salient regions to a greater extent than other image aspects, wherein each of the first image, the second image, and third image is multiscopically distinct.

6. A non-transitory computer-readable medium containing a program which, when executed, performs an operation for multiscopic video augmentation based on non-linear warp functions, the operation comprising:
receiving a multiscopic video frame comprising at least a first image and a second image, at least one image of which has a first set of pixels and a second set of pixels;
analyzing the first image and the second image of the multiscopic video frame in order to determine a set of image characteristics including a set of sparse disparities between the first and second images;
determining, based on the set of image characteristics, a mapping function comprising a non-linear warp function configured to, in an image domain, warp the first set of pixels to a greater extent than the second set of pixels; and
generating, by operation of one or more computer processors when executing the program, at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

7. The non-transitory computer-readable medium of claim 6 , wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

8. The non-transitory computer-readable medium of claim 7 , wherein the saliency map is determined by analyzing a predefined set of image attributes based on a quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation.

9. The non-transitory computer-readable medium of claim 8 , wherein the set of image characteristics further includes an edge map of at least one of the first image and the second image, wherein the non-linear warp function is determined by optimizing an energy equation based on the set of image characteristics and a set of associated weights.

10. The non-transitory computer-readable medium of claim 9 , wherein the multiscopic video frame comprises a stereoscopic video frame, wherein the third image is generated further based on at least one of interpolation and extrapolation;
wherein the set of sparse disparities between the first image and second image are determined based on feature matching and optical flow analysis;
wherein the saliency map is determined by analyzing a predefined set of image attributes based on quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation;
wherein the edge map is determined based on edge detection and vertical line detection, wherein the edge detection is based on a Canny algorithm, wherein the vertical line detection is based on a Hough transform;
wherein the determined mapping function is configured to, when generating the third image, preserve vertical edges and salient regions to a greater extent than other image aspects, wherein each of the first image, the second image, and third image is multiscopically distinct.

11. A system of multiscopic video augmentation based on non-linear warp functions, the system comprising:
one or more computer processors;
a memory containing a program which, when executed by the one or more computer processors, performs an operation comprising:
receiving a multiscopic video frame comprising at least a first image and a second image, at least one image of which has a first set of pixels and a second set of pixels;
analyzing the first image and the second image of the multiscopic video frame in order to determine a set of image characteristics including a set of sparse disparities between the first and second images;
determining, based on the set of image characteristics, a mapping function comprising a non-linear warp function configured to, in an image domain, warp the first set of pixels to a greater extent than the second set of pixels; and
generating at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

12. The system of claim 11 , wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

13. The system of claim 12 , wherein the saliency map is determined by analyzing a predefined set of image attributes based on a quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transforma

Assignees

• Stefanoski Nikolce
• Smolic Aljoscha
• Lang Manuel
• Farré Miquel À
• Hornung Alexander
• Fricke Pedro Christian Espinosa
• Wang Oliver
• Disney Entpr Inc
• Eth Zurich (Eidgenoessische Technische Hochschule Zurich)

Inventors

• Stefanoski Nikolce
• Smolic Aljoscha
• Lang Manuel
• Farré Miquel À
• Hornung Alexander
• Fricke Pedro Christian Espinosa
• Wang Oliver

Classifications

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• H04N13/122 Primary

  Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) · CPC title

• H04N13/0018 Primary

  Electricity · mapped topic