Video capture, processing, calibration, computational fiber artifact removal, and light-field pipeline

Patent metadata

Field	Value
Publication number	US-10565734-B2
Application number	US-201715451831-A
Country	US
Kind code	B2
Filing date	Mar 7, 2017
Priority date	Apr 15, 2015
Publication date	Feb 18, 2020
Grant date	Feb 18, 2020

Abstract

An image capture system includes a plurality of image sensors arranged in a pattern such that gaps exist between adjacent image sensors of the plurality of image sensors. Each of the image sensors may be configured to capture sensor image data. The image capture system may also have a main lens configured to direct incoming light along an optical path, a microlens array positioned within the optical path, and a plurality of tapered fiber optic bundles. Each tapered fiber optic bundle may have a leading end positioned within the optical path, and a trailing end positioned proximate one of the image sensors. The leading end may have a larger cross-sectional area than the trailing end. Sensor data from the image sensors may be combined to generate a single light-field image that is substantially unaffected by the gaps.

First claim

What is claimed is:

1. A method for capturing and processing image data, the method comprising:
receiving incoming light along an optical path in an image capture device, the image capture device comprising:
a plurality of image sensors arranged in a pattern such that gaps exist between adjacent image sensors of the plurality of image sensors, wherein the plurality of image sensors includes a first image sensor and a second image sensor;
a main lens;
a microlens array;
a plurality of tapered fiber optic bundles, each of which comprises a leading end positioned within the optical path, and a trailing end positioned proximate one of the plurality of image sensors, the leading end having a larger cross-sectional area than the trailing end, wherein the plurality of tapered fiber optic bundles includes:
a first tapered fiber optic bundle of a first length positioned proximate the first image sensor;
a second tapered fiber optic bundle of a second length longer than the first length positioned proximate the second image sensor; and
a beam splitter to direct light from the main lens to the plurality of tapered fiber optic bundles, and wherein the tapered fiber optic bundles are arranged along a first plane and a second plane;

with the main lens, directing the incoming light along the optical path;
directing the incoming light through the microlens array;
directing at least some of the incoming light through the tapered fiber optic bundles including the first and second tapered fiber optic bundles to the image sensors;
with the image sensors including the first and second image sensors, receiving the incoming light to generate image sensor data; and
combining the image sensor data generated by the image sensors including the first and second image sensors to define a single light-field image such that the single light-field image is substantially unaffected by the gaps.

2. The method of claim 1 , further comprising:
directing some of the incoming light to a preview camera; and:
with the preview camera, generating a preview image representative of the single light-field image.

3. The method of claim 2 , further comprising, prior to directing the incoming light along the optical path, calibrating at least one of the image capture device and the preview camera to align an attribute of the preview camera with a corresponding attribute of the image capture device, wherein the attribute is selected from:
an ISO setting;
a focus setting;
a zoom setting; and
an image crop setting.

4. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to mitigate differences in sensitivity between the image sensors.

5. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to mitigate at least one of:
hex noise patterns existing within the tapered fiber optic bundles;
fiber noise patterns existing within the tapered fiber optic bundles; and
fault line defects existing within the tapered fiber optic bundles.

6. The method of claim 1 , further comprising, prior to combining the image sensor data to define the single light-field image, transmitting the image sensor data to a plurality of servers;
wherein combining the image sensor data to define the single light-field image comprises combining the image sensor data at the plurality of servers.

7. The method of claim 1 , further comprising, after combining the image sensor data to define the single light-field image, performing one or more light-field image processing steps on the single light-field image.

8. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path:
calibrating the image capture device to perform at least one of:
ISO calibration;
sensor dark current calibration;
dark fiber artifact removal;
sensor cropping;
grid warp calibration;
fault line artifact removal; and
after calibration of the image capture device, securing the microlens array relative to the image sensors.

9. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to perform at least one of:
calibrating an aperture versus a focal length of the main lens; and
capturing a white image for demodulation.

10. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to perform 2D geometry distortion correction by:
with the image capture device, capturing a calibration image of a checkerboard;
running a corner detection algorithm to detect corners of the checkerboard in the calibration image; and
running a grid fitting algorithm to define a reference grid fitted to the corners to generate vectors that can be used to generate an undistorted image from a distorted image.

11. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device to perform 2D geometry distortion correction by:
with the image capture device, capturing a plurality of calibration images of a checkerboard, with the checkerboard incrementally offset in each successive image;
generating a plurality of product images by performing a pixel-by-pixel multiplication of one vertical line image and one horizontal line on each of the plurality of calibration images;
for each of the product images, calculating a center-of-mass of each of a plurality of box-shaped areas to generate a grid; and
for each of the grids, applying a global optimization routine to identify a reference grid that fits the grid to generate vectors that can be used to generate an undistorted image from a distorted image.

12. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by carrying out fault line artifact removal.

13. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by performing tapered fiber optic bundle image alignment to generate geometry transform data;
wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises using the geometry transform data to stitch images of the image sensor data together.

14. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by generating a plurality of masks;
wherein combining the image sensor data generated by the image sensors to define the single light-field image comprises:
detecting seams between images of the image sensor data by finding center lines of each overlapping subregion between masks of adjacent image sensors; and
blending pixels proximate the seams.

15. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by performing dark frame calibration for each of the image sensors.

16. The method of claim 1 , further comprising, prior to directing the incoming light along the optical path, calibrating the image capture device by performing flat-field calibration for each of the image sensors.

17. The method of claim 1 , further comprising, prior to combining the image sensor data generated by the image sensors to define the single light-field image, processin

Assignees

• Google Llc

Inventors

• Bevensee Brendan
• Du Tingfang
• Karafin Jon
• Merritt Joel
• Petrovich Duane
• Spor Gareth

Classifications

• G06T7/521

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• H04N17/002

  for television cameras · CPC title

• G02B3/0056 Primary

  arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses (G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085) · CPC title

• H04N13/282

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• G02B6/08

  with fibre bundle in form of plate · CPC title