Generating Images from Light Fields Utilizing Virtual Viewpoints

Patent metadata

Field	Value
Publication number	US-2016255333-A1
Application number	US-201514876024-A
Country	US
Kind code	A1
Filing date	Oct 6, 2015
Priority date	Sep 28, 2012
Publication date	Sep 1, 2016
Grant date	—

Abstract

Systems and methods for the synthesis of light field images from virtual viewpoints in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system includes a processor and a memory configured to store captured light field image data and an image manipulation application, wherein the captured light field image data includes image data, pixel position data, and a depth map, and wherein the image manipulation application configures the processor to obtain captured light field image data, determine a virtual viewpoint for the captured light field image data, where the virtual viewpoint includes a virtual location and virtual depth information, compute a virtual depth map based on the captured light field image data and the virtual viewpoint, and generate an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map.

First claim

1 . A system configured to synthesize images using captured light field image data, comprising:
a processor; and a memory connected to the processor and configured to store captured light field image data and an image manipulation application; wherein the captured light field image data comprises image data, pixel position data, and a depth map; wherein the depth map comprises depth information for one or more pixels in the image data; and wherein the image manipulation application configures the processor to:
obtain captured light field image data;
determine a virtual viewpoint for the captured light field image data based on the pixel position data and the depth map for the captured light field image data, where the virtual viewpoint comprises a virtual location and virtual depth information;
compute a virtual depth map based on the captured light field image data and the virtual viewpoint; and
generate an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map, where the generated image comprises a plurality of pixels selected from the image data based on the pixel position data and the virtual depth map.

2 . The system of claim 1 , wherein the virtual viewpoint corresponds to a focal plane in an array camera utilized to create the captured light field image data.

3 . The system of claim 1 , wherein:
the captured light field image data further comprises a reference viewpoint within the captured light field image data; and the virtual viewpoint is a separate viewpoint within the captured light field image data from the reference viewpoint.

4 . The system of claim 3 , wherein:
the captured light field image data was captured by an array camera having an imager array comprising a plurality of imagers; and the reference viewpoint corresponds to the viewpoint of a first imager within the imager array in the array camera.

5 . The system of claim 4 , wherein the virtual viewpoint corresponds to the viewpoint of a second imager within the imager array, where the second imager is separate from the first imager.

6 . The system of claim 4 , wherein the virtual viewpoint is a viewpoint that does not correspond to the viewpoint of any of the imagers within the imager array.

7 . The system of claim 4 , wherein the virtual viewpoint is selected from a position selected from the group consisting of in front of the imager array and behind the imager array.

8 . The system of claim 1 , wherein the image manipulation application further configures the processor to generate an image from the perspective of the virtual viewpoint by projecting pixels from the captured light field image data based on the pixel position data and the depth map, where the projected pixels are described in the image data and the depth map.

9 . The system of claim 8 , wherein:
the captured light field image data further comprises occluded pixel information describing pixels not visible from a reference viewpoint of the captured light field image data; and the projected pixels include at least one occluded pixel in the occluded pixel information that is visible from the perspective of the virtual viewpoint.

10 . The system of claim 8 , wherein:
at least one projected pixel in the generated image is not described in the image data, the pixel position data, and the depth map; and the image manipulation application further configures the processor to generate the at least one projected pixel by resampling the image data, the pixel position data, and the depth map.

11 . The system of claim 8 , wherein a pinhole camera model is utilized to project pixels within the generated image based on light rays projecting from the virtual viewpoint, where each projected pixels is associated with at least one of the projected light rays.

12 . The system of claim 11 , wherein projected pixel depth information is determined for at least one pixel in the generated image based on the depth map, the virtual viewpoint, and the light rays associated with the projected pixel.

13 . The system of claim 12 , wherein the depth information for a projected pixel is based on minimizing the variance for the projected pixel across the image data within the captured light field image data.

14 . The system of claim 11 , wherein the image manipulation application further configures the processor to combine projected pixels having the same location within the generated image.

15 . The system of claim 14 , wherein the pixels are combined based on the weighted average of the pixels, where the weighted average of the pixels is the inverse of the distance from the imager from which the projected pixel originated to the virtual viewpoint.

16 . The system of claim 1 , further comprising an input device configured to obtain input data indicative of a position within the captured light field image data.

17 . The system of claim 16 , wherein the input device is a touchscreen interface.

18 . The system of claim 16 , wherein the input device is a sensor configured to obtain spatial location information.

19 . The system of claim 16 , wherein the input device is a camera configured to obtain input data selected from the group consisting of head tracking data and gaze tracking data.

20 . The system of claim 16 , wherein the virtual viewpoint is selected based on the input data.

21 - 23 . (canceled)

Assignees

• Pelican Imaging Corp

Inventors

• Nisenzon Semyon
• Jain Ankit K

Classifications

• H04N23/13

  with multiple sensors · CPC title

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• G06T7/593

  from stereo images · CPC title

• G06T7/596

  from three or more stereo images · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title