Systems and methods for reducing motion blur in images or video in ultra low light with array cameras

Patent metadata

Field	Value
Publication number	US-9578259-B2
Application number	US-201414776553-A
Country	US
Kind code	B2
Filing date	Mar 12, 2014
Priority date	Mar 14, 2013
Publication date	Feb 21, 2017
Grant date	Feb 21, 2017

Abstract

Systems and methods for reducing motion blur in images or video in ultra low light with array cameras in accordance with embodiments of the invention are disclosed. In one embodiment, a method for synthesizing an image from multiple images captured using an array camera includes capturing image data using active cameras within an array camera, where the active cameras are configured to capture image data and the image data includes pixel brightness values that form alternate view images captured from different viewpoints, determining sets of corresponding pixels in the alternate view images where each pixel in a set of corresponding pixels is chosen from a different alternate view image, summing the pixel brightness values for corresponding pixels to create pixel brightness sums for pixel locations in an output image, and synthesizing an output image from the viewpoint of the output image using the pixel brightness sums.

First claim

What is claimed is:

1. A method for synthesizing an image from multiple images captured from different viewpoints using an array camera in low light conditions, the method comprising:
capturing image data using a plurality of active cameras within an array camera, where the plurality of active cameras are configured to capture image data within the same spectral band and the image data captured by the active cameras comprises pixel brightness values that form a reference image and a plurality of alternate view images captured from different viewpoints;
applying geometric shifts to shift the plurality of alternate view images to the viewpoint of the reference image using the processor configured by software;
summing the pixel brightness values for pixels in the reference image with pixel brightness values for corresponding pixels in the alternate view images to create pixel brightness sums for the pixel locations in the reference image using the processor configured by software; and
synthesizing an output image from the viewpoint of the reference image using image data comprising the pixel brightness sums for the pixel locations in the reference image using the processor configured by software.

2. The method of claim 1 , wherein applying geometric shifts to shift the plurality of alternate view images to the viewpoint of the reference image further comprises applying scene independent geometric shifts to the alternate view images to compensate for distortions due to physical characteristics of the plurality of active cameras that captured the alternate view images.

3. The method of claim 2 , further comprising:
performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to the alternate view images by comparing the reference image and the alternate view images;
wherein applying geometric shifts to shift the plurality of alternate view images to the viewpoint of the reference image further comprises applying scene dependent geometric shifts to the plurality of alternate view images to compensate for parallax.

4. The method of claim 3 , further comprising:
identifying pixels in the alternate view images that are occluded in the reference image using the processor configured by software; and
leaving occluded pixels out when summing the pixel brightness values for pixels in the reference image with pixel brightness values for corresponding pixels in the alternate view images using the processor configured by software.

5. The method of claim 2 , further comprising:
performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to at least a portion of the pixels in the alternate view images by comparing the reference image and the alternate view images; and
when parallax detection identifies at least one pixel within a threshold distance of the reference viewpoint, applying scene dependent geometric shifts to the plurality of alternate view images to compensate for parallax.

6. The method of claim 2 , further comprising:
performing parallax detection using the processor configured by software to identify scene dependent geometric shifts to apply to at least a portion of the pixels in the alternate view images by comparing the reference image and the alternate view images; and
when parallax detection determines that a pixel from the reference viewpoint has a depth within a specified depth of field, applying scene dependent geometric shifts to corresponding pixels in the alternate view images to compensate for parallax.

7. The method of claim 6 , further comprising receiving user input specifying a depth of field via a user interface using the processor configured by software.

8. The method of claim 6 , further comprising automatically determining a specified depth of field based upon a depth of an object within a region of interest using the processor configured by software.

9. The method of claim 2 , wherein applying geometric shifts to shift the plurality of alternate view images to the viewpoint of the reference image further comprises applying a fixed parallax shift to the plurality of alternate view images.

10. The method of claim 9 , further comprising determining the fixed parallax shift based upon user input specifying a depth received via a user interface using the processor configured by software.

11. The method of claim 9 , further comprising automatically determining a fixed parallax shift based upon a depth of an object within a region of interest using the processor configured by software.

12. The method of claim 11 , wherein automatically determining a fixed parallax shift based upon a depth of an object within a region of interest further comprises:
calculating a depth map for a region of interest;
generating a histogram of depths in the region of interest; and
determining the depth of an object within the region of interest as the median depth of the region of interest.

13. The method of claim 1 , further comprising:
capturing a second set of image data using the plurality of active cameras and synthesizing a second output image using the processor configured by software;
calculating motion compensation vectors for the second output image using the processor configured by software;
applying motion compensation shifts to shift the second output image to the viewpoint of the output image using the processor configured by software;
summing the pixel brightness values for pixels in the output image with pixel brightness values for corresponding pixels in the second output image to create pixel brightness sums for the pixel locations in the output image using the processor configured by software; and
synthesizing a motion compensated output image from the viewpoint of the reference image using the pixel brightness sums for the pixel locations in the output image using the processor configured by software.

14. The method of claim 1 , wherein the plurality of active cameras that capture the reference image and the alternate view images form a first subset of cameras and the method further comprises:
capturing image data using a second subset of active cameras within the array camera, where the second subset of active cameras are configured to capture image data within the same spectral band as the first subset of cameras and the image data captured by the active cameras comprises pixel brightness values that form a second reference image and a second set of alternate view images captured from different viewpoints;
applying geometric shifts to shift the second set of alternate view images to the viewpoint of the second reference image using the processor configured by software;
summing the pixel brightness values for pixels in the second reference image with pixel brightness values for corresponding pixels in the second set of alternate view images to create pixel brightness sums for the pixel locations in the second reference image using a processor configured by software;
synthesizing an alternate view output image from the viewpoint of the second reference image using the pixel brightness sums for the pixel locations in the second reference image using the processor configured by software; and
synthesizing a high resolution image using the processor configured by software to perform a super resolution process based upon the output image and the alternate view output image.

15. The method of claim 1 , wherein the array camera comprises cameras that capture image data within different spectral bands.

16. The method of claim 15 , wherein

Assignees

• Pelican Imaging Corp
• Fotonation Cayman Ltd

Inventors

• Molina Gabriel

Classifications

• H04N23/70

  Circuitry for compensating brightness variation in the scene · CPC title

• H04N23/68

  for stable pick-up of the scene, e.g. compensating for camera body vibrations · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• H04N23/45 Primary

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/682

  Vibration or motion blur correction · CPC title