High resolution free-view interpolation of planar structure

Patent metadata

Field	Value
Publication number	US-9824486-B2
Application number	US-201414572524-A
Country	US
Kind code	B2
Filing date	Dec 16, 2014
Priority date	Dec 16, 2013
Publication date	Nov 21, 2017
Grant date	Nov 21, 2017

Abstract

A method and system is proposed to create a generative model to interpolate any view of a planar scene given a sequence of reference views and a synthesis view that is optimized by the marginalization of photometric regulation, and geometric registration parameters. According to one aspect of the claimed subject matter, a technique is proposed to combine information from varying input camera poses. Planar homography based image super resolution in free view interpolation for planar structure is applied to the combined information. Non-redundant information is combined in such a manner that the high resolution and free view problems in traditional 2D based image-based rendering techniques are overcome.

First claim

What is claimed is:

1. A computer implemented method of rendering a free view image of a captured scene, the method comprising:
accessing a plurality of input images stored in a memory of a computing device and corresponding to a plurality of reference viewpoints, wherein an input image represents a planar image of the captured scene at a respective reference viewpoint;
receiving a requested viewpoint;
generating, by a processor of the computing device, an initial synthesis image based on the plurality of input images using a homography transformation of one of the input images and average color intensity values of pixels included in the plurality of input images;
calculating, by the processor of the computing device, geometric registration parameters and photometric regulation parameters of the plurality of input images, wherein the geometric registration parameters comprise at least one planar homography transformation matrix between the plurality of input images;
calculating the at least one planar homographic transformation matrix by extracting and matching scale-invariant feature transform (SIFT) points in the initial synthesis image and the plurality of input images in accordance with direct linear transformation (DLT) using a random sample consensus (RANSAC) method;
calculating photometric regulation parameters in accordance with a linear regression method; and
generating, by the processor of the computing device, a new synthesis image representing a planar image of the captured scene in the requested viewpoint to refine the initial synthesis image based on an interpolation of the plurality of input images, the geometric registration parameters, and the photometric regulation parameters.

2. The computer implemented method of claim 1 , wherein calculating the geometric registration parameters comprises determining a positional relativity between pixels comprised in the initial synthesis image and pixels comprised in the plurality of input images.

3. The computer implemented method of claim 2 , wherein determining the positional relativity comprises generating the at least one planar homography transformation matrix relating the plurality of input images.

4. The computer implemented method of claim 1 , wherein the photometric regulation parameters corresponds to a relationship between the average color intensity values of the pixels comprised in the initial synthesized image with intensity values of pixels comprised in the plurality of input images.

5. The computer implemented method of claim 1 , wherein the new synthesis image comprises initial photometric regulation parameters, wherein the initial photometric regulation parameters are calculated based on the average color intensity values of the pixels included in the plurality of input images.

6. The computer implemented method of claim 1 , further comprising:
iteratively re-calculating the photometric regulation parameters based on the new synthesis image;
iteratively re-calculating the geometric registration parameters based on the new synthesis image; and
re-estimating the new synthesis image based on the re-calculated photometric regulation parameters and the re-calculated geometric registration parameters.

7. The computer implemented method of claim 6 , wherein the new synthesis image is calculated iteratively according to a Bayesian method.

8. The computer implemented method of claim 6 , wherein a distribution of the geometric registration parameters is calculated according to a property of Lie group.

9. The computer implemented method of claim 6 , wherein re-calculating the photometric regulation parameters and re-calculating the geometric registration parameters of pixels are performed substantially contemporaneously.

10. The computer implemented method of claim 1 , wherein the free view image has a greater resolution than each input image, and wherein the requested viewpoint is substantially different from the reference viewpoints associated with the plurality of input images in full 6 degrees of freedom (6DOF).

11. The computer implemented method of claim 10 , wherein the at least one planar homography transformation matrix comprises a full 6DOF planar homography transformation matrix.

12. The computer implemented method of claim 1 , wherein the at least one planar homographic transformation matrix comprises at least one 3×3 real matrix with unit determinant and 8-degrees of freedom (DOF) corresponding to a Lie group.

13. The computer implemented method of claim 1 , further comprising calculating a joint probability distribution of the at least one planar homographic transformation matrix, the photometric regulation parameters, and the initial synthesis image, wherein generating the initial synthesis image further comprises searching the photometric regulation parameters and the geometric registration parameters based on the joint probability distribution.

14. The computer implemented method of claim 13 , further comprising:
calculating an object function of the joint probability distribution; and
optimizing the object function in accordance with a bounded quasi-Newton method.

15. The computer implemented method of claim 1 , wherein calculating the geometric registration parameters comprises:
calculating an initial planar homography transformation matrix between the plurality of input images using scale-invariant feature transform (SIFT) matching;
generating the initial synthesis image using the initial planar homography transformation matrix over a selected input image; and
deriving a new planar homography transformation matrix between the initial synthesis image and the plurality of input images.

16. The computer implemented method of claim. 1 , wherein reference viewpoints associated with the plurality of input images are substantially different in view angles and view depths.

17. The computer implemented method of claim 1 , wherein calculating the geometric registration parameters comprises adding Gaussian independent identically distributed (i.i.d.) noise to each dimension of a homogeneous coordinate.

18. The computer implemented method of claim 1 , wherein the plurality of input images comprises at least one image of a group of images consisting of:
an image captured by an imaging device;
a computer simulated image;
a cropped portion of an image;
a segmented portion of an image; or
an image extracted from a video file.

19. The computer implemented method of claim 1 , wherein generating the synthesis image comprises generating the synthesis image in accordance with a super resolution (SR) method.

20. The computer implemented method of claim 1 , wherein the free view image corresponds to at least one of a zoomed-in image, a zoomed-out image, and a rotated image of the captured scene.

21. A system for rendering a high resolution free view image of a captured scene in an image capture device, the system comprising:
the image capture device comprising:
a lens; and
a plurality of shutters,
wherein the image capture device is configured to generate a plurality of input images corresponding to a plurality of reference viewpoints, and
wherein an input image represents a planar image of the captured scene at a respective reference viewpoint;

a memory, communicatively coupled to the image capture device, and configured to store image data corresponding to the plurality of input images and a plurality of programmed instructions; and
a processor, communicatively coupled to the memory, and configured to ex

Assignees

• Futurewei Technologies Inc

Inventors

• Hu Jie
• Chen Chang Wen
• Zhang Dongqing
• Yu Hong Heather

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N5/2628

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• H04N5/76

  Television signal recording · CPC title

• G06T15/205 Primary

  Image-based rendering · CPC title