Capturing light-field volume image and video data using tiled light-field cameras

Patent metadata

Field	Value
Publication number	US-2016307372-A1
Application number	US-201615084326-A
Country	US
Kind code	A1
Filing date	Mar 29, 2016
Priority date	Apr 15, 2015
Publication date	Oct 20, 2016
Grant date	—

Abstract

A capture system may capture light-field data representative of an environment for use in virtual reality, augmented reality, and the like. The system may have a plurality of light-field cameras arranged to capture a light-field volume within the environment, and a processor. The processor may use the light-field volume to generate a first virtual view depicting the environment from a first virtual viewpoint. The light-field cameras may be arranged in a tiled array to define a capture surface with a ring-shaped, spherical, or other arrangement. The processor may map the pixels captured by the image sensors to light rays received in the light-field volume, and store data descriptive of the light rays in a coordinate system representative of the light-field volume.

First claim

What is claimed is:

1 . A system comprising:
a plurality of light-field cameras arranged in a tiled array to capture a light-field volume within an environment; and a processor configured to use the light-field volume to generate a first virtual view depicting the environment from a first virtual viewpoint.

2 . The system of claim 1 , wherein the light-field cameras are arranged such that outward-facing elements of the light-field cameras define a physical capture surface larger than the light-field volume;
wherein the physical capture surface encircles the light-field volume.

3 . The system of claim 2 , wherein the processor is further configured to interpolate between portions of the light-field volume captured by the light-field cameras to generate additional data to fill holes in the light-field volume resulting from gaps between the light-field cameras.

4 . The system of claim 1 , wherein at least a subset of the light-field cameras are arranged in a ring array to define a generally cylindrical capture surface that captures the light-field volume.

5 . The system of claim 4 , wherein the light-field cameras are arranged in multiple layers in the ring array to define the generally cylindrical capture surface.

6 . The system of claim 4 , wherein at least one of light-field cameras is positioned on an axis of the generally cylindrical capture surface and is oriented substantially perpendicularly to the light-field cameras of the subset.

7 . The system of claim 1 , wherein the light-field cameras are arranged in a spherical array to define a generally spherical capture surface that captures the light-field volume.

8 . The system of claim 1 , wherein each of the light-field cameras comprises an image sensor and a microlens array that projects light from the light-field volume onto the image sensor.

9 . The system of claim 8 , wherein each of the light-field cameras further comprises a tapered fiber optic bundle comprising a leading end facing toward the environment and a trailing end positioned proximate the image sensor.

10 . The system of claim 1 , wherein:
each of the light-field cameras comprises an objective lens through which light from the light-field volume enters the light-field camera; and the objective lenses of the light-field cameras are compactly arranged to minimize gaps between the objective lenses.

11 . The system of claim 1 , wherein:
each of the light-field cameras comprises an objective lens through which light from the light-field volume enters the light-field camera; and each of the objective lenses comprises a double Gauss lens.

12 . The system of claim 1 , further comprising a motor that rotates the light-field cameras relative to the environment.

13 . The system of claim 1 , wherein each of the light-field cameras comprises an objective lens through which the light-field volume enters the light-field camera and an image sensor that captures the light-field volume;
and wherein the system further comprises a motor that rotates the image sensors of the light-field cameras relative to the objective lenses of the light-field cameras.

14 . The system of claim 1 , wherein each of the light-field cameras comprises an array light-field camera.

15 . The system of claim 1 , wherein:
each of the light-field cameras comprises an image sensor that captures pixels representative of the environment; and the processor is further configured to map the pixels to light rays received in the light-field volume, and to store data descriptive of the light rays in a coordinate system representative of the light-field volume.

16 . The system of claim 15 , wherein:
the coordinate system comprises a polar coordinate system; and the light rays are recorded in terms of a first angle of intersection with an inner sphere, a second angle of intersection with the inner sphere, a first angle of intersection with an outer sphere, and a second angle of intersection with the outer sphere.

17 . The system of claim 15 , wherein the processor is further configured to generate the first virtual view by:
positioning a virtual lens and a virtual image sensor on the coordinate system; and modeling passage of the light rays through the virtual lens to the virtual image sensor.

18 . The system of claim 1 , wherein:
the processor is further configured to use the light-field volume to generate a second virtual view depicting the environment from a second virtual viewpoint; and the second virtual viewpoint is spaced apart from the first virtual viewpoint such that the first virtual view and the second virtual view cooperate to simulate viewing of the environment by a user.

19 . A system comprising:
a plurality of center view cameras arranged in at least a first circular pattern around an axis, wherein each of the center view cameras has a field-of-view generally perpendicular to the axis; a plurality of upper view cameras arranged in at least a second circular pattern around the axis, wherein each of the upper view cameras has a field-of-view oriented generally upward; and a plurality of lower view cameras arranged in at least a third circular pattern around the axis, wherein each of the lower view cameras has a field-of-view oriented generally downward; and wherein the first circular pattern is interleaved with at least one of the second circular pattern and the third circular pattern to define a first ring in which each of the center view cameras is positioned between two elements selected from the group consisting of the upper view cameras and the lower view cameras.

20 . The system of claim 19 , further comprising a processor configured to use image data captured by the center view cameras, the upper view cameras, and the lower view cameras from within an environment to generate a first virtual view depicting the environment from a first virtual viewpoint.

21 . The system of claim 19 , wherein the first circular pattern is interleaved with the second circular pattern and the third circular pattern to define the first ring such that, in the first ring, each of the center view cameras is positioned between one of the upper view cameras and one of the lower view cameras.

22 . The system of claim 19 , wherein:
the first circular pattern is interleaved with the second circular pattern to define the first ring such that, in the first ring, each of the center view cameras is positioned between two of the upper view cameras; and the center view cameras are further arranged in a fourth circular pattern interleaved with the third circular pattern to define a second ring such that, in the second ring, each of the center view cameras is positioned between two of the lower view cameras.

23 . The system of claim 19 , wherein:
each of the center view cameras comprises a center view camera field-of-view; each of the upper view cameras comprises an upper view camera field-of-view; each of the lower view cameras comprises a lower view camera field-of-view; and the upper view camera field-of-view and the lower view camera field-of-view are larger than the center view camera field-of-view.

24 . A method of generating one or more virtual viewpoints of an environment, the method comprising:
at a plurality of light-field cameras arranged in a tiled array, capturing a light-field volume within the environment; and at a processor, using the light-field volume to generate a first virtual view depi

Assignees

• Lytro Inc

Inventors

• Pitts Colvin
• Ha Yonggang
• Houck William

Classifications

• H04N23/957

  Light-field or plenoptic cameras or camera modules · CPC title

• H04N13/0228

  Electricity · mapped topic

• H04N13/0011

  Electricity · mapped topic

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N13/0282

  Electricity · mapped topic