Multi-view back-projection to a light-field

Patent metadata

Field	Value
Publication number	US-10354399-B2
Application number	US-201715605037-A
Country	US
Kind code	B2
Filing date	May 25, 2017
Priority date	May 25, 2017
Publication date	Jul 16, 2019
Grant date	Jul 16, 2019

Abstract

Dense light-field data can be generated from image data that does not include light-field data, or from image data that includes sparse light-field data. In at least one embodiment, the source light-field data may include one or more sub-aperture images that may be used to reconstruct the light-field in denser form. In other embodiments, the source data can take other forms. Examples include data derived from or ancillary to a set of sub-aperture images, synthetic data, or captured image data that does not include full light-field data. Interpolation, back-projection, and/or other techniques are used in connection with source sub-aperture images or their equivalents, to generate dense light-field data.

First claim

What is claimed is:

1. A method for generating a reconstructed light-field, the method comprising:
from a storage device, retrieving a first image depicting a light-field from a first viewpoint;
in a processor, obtaining a first value of interest pertinent to the first image; and
in the processor, using the first value of interest to generate the reconstructed light-field by extrapolating corresponding values, corresponding to the first value of interest, for a first plurality of other images;
wherein:
the first plurality of other images comprises at least a second image depicting the light-field from a second viewpoint, and a third image depicting the light-field from a third viewpoint; and
using the first value of interest to extrapolate the corresponding values comprises:
based on relative positions of the first viewpoint and the second viewpoint, extrapolating a second value of interest pertinent to the second image; and
based on relative positions of the first viewpoint and the third viewpoint, extrapolating a third value of interest pertinent to the third image,

wherein:
the second viewpoint is displaced from the first viewpoint along a second direction; and
the third viewpoint is displaced form the first viewpoint along a third direction opposite to the second direction,
wherein:
using the first value of interest to generate the reconstructed light-field further comprises extrapolating corresponding values, corresponding to the first value of interest, for a second plurality of other images;
the second plurality of other images comprises at least a fourth image depicting the light-field from a fourth viewpoint, and a fifth image depicting the light-field from a fifth viewpoint;
the fourth viewpoint is displaced from the first viewpoint along a fourth direction perpendicular to the second direction and the third direction;
the fifth viewpoint is displaced from the first viewpoint along a fifth direction opposite to the fourth direction; and
using the first value of interest to extrapolate the corresponding values comprises:
based on relative positions of the first viewpoint and the fourth viewpoint, extrapolating a fourth value of interest pertinent to the fourth image; and
based on relative positions of the first viewpoint and the fifth viewpoint, extrapolating a fifth value of interest pertinent to the fifth image.

2. The method of claim 1 , wherein the first value of interest comprises a depth of at least one object depicted in the first image.

3. The method of claim 2 , wherein:
extrapolating the corresponding values, for the first plurality of other images comprises:
back-projecting a second depth map for the second image; and
back-projecting a third depth map for the third image; and

the method further comprises using weighted averages of at least the second depth map and the third depth map to generate a reconstructed depth-field for objects within the reconstructed light-field.

4. The method of claim 1 , wherein the first value of interest comprises an image value of at least one portion of the first image.

5. The method of claim 4 , wherein extrapolating the corresponding values, for the first plurality of other images comprises:
back-projecting a second light-field for the second image;
back-projecting a third light-field for the third image; and
using weighted averages of at least the second light-field and the third light-field to generate the reconstructed light-field.

6. The method of claim 4 , wherein extrapolating the corresponding values, for the first plurality of other images comprises:
generating a reconstructed depth-field for objects within the reconstructed light-field; and
using the reconstructed depth-field to back-project image values for the reconstructed light-field.

7. The method of claim 1 , further comprising, after generation of the reconstructed light-field, using the reconstructed light-field to project a new image from a new viewpoint.

8. The method of claim 1 , wherein:
the first image comprises a first sub-aperture image projected from light-field data captured with a light-field camera;
the second image comprises a second sub-aperture image projected from the light-field data; and
the third image comprises a third sub-aperture image projected from the light-field data.

9. The method of claim 1 , wherein:
the first image comprises first image data derived from a first sub-aperture image projected from light-field data captured with a light-field camera;
the second image comprises second image data derived from a second sub-aperture image projected from the light-field data; and
the third image comprises third image data derived from a third sub-aperture image projected from the light-field data.

10. The method of claim 1 , wherein each of the first image, the second image, and the third image consists exclusively of data generated without involving capture of image data by a camera.

11. The method of claim 1 , wherein each of the first image, the second image, and the third image comprises a conventional image captured with a camera that is not a light-field camera.

12. A non-transitory computer-readable medium for generating a reconstructed light-field, the non-transitory computer-readable medium comprising instructions stored thereon, that when executed by a processor, perform the steps of:
causing a storage device to retrieve a first image depicting a light-field from a first viewpoint;
obtaining a first value of interest pertinent to the first image; and
using the first value of interest to generate the reconstructed light-field by extrapolating corresponding values, corresponding to the first value of interest, for a first plurality of other images;
wherein:
the first plurality of other images comprises at least a second image depicting the light-field from a second viewpoint, and a third image depicting the light-field from a third viewpoint; and
using the first value of interest to extrapolate the corresponding values comprises:
based on relative positions of the first viewpoint and the second viewpoint, extrapolating a second value of interest pertinent to the second image; and
based on relative positions of the first viewpoint and the third viewpoint, extrapolating a third value of interest pertinent to the third image,
wherein:
the second viewpoint is displaced from the first viewpoint along a second direction;
the third viewpoint is displaced form the first viewpoint along a third direction opposite to the second direction;
using the first value of interest to generate the reconstructed light-field further comprises extrapolating corresponding values, corresponding to the first value of interest, for a second plurality of other images;
the second plurality of other images comprises at least a fourth image depicting the light-field from a fourth viewpoint, and a fifth image depicting the light-field from a fifth viewpoint;
the fourth viewpoint is displaced from the first viewpoint along a fourth direction perpendicular to the second direction and the third direction;
the fifth viewpoint is displaced from the first viewpoint along a fifth direction opposite to the fourth direction; and
using the first value of interest to extrapolate the corresponding values comprises:
based on relative positions of the first viewpoint and the fourth viewpoint, extrapolating a fourth value of interest pertinent to the fourth image; and
based on relative positions of the first viewpoint and the fifth viewpoint, extrapolating a fifth value of interest pertinent to the fifth image.

13. The non-transitory computer-readable medium of claim 12 , wh

Assignees

• Google Llc

Inventors

• Wang Zejing
• Nonn Thomas

Classifications

• H04N23/957

  Light-field or plenoptic cameras or camera modules · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• H04N13/218

  using spatial multiplexing · CPC title

• G06T7/557 Primary

  from light fields, e.g. from plenoptic cameras · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title