Array cameras incorporating optics with modulation transfer functions greater than sensor Nyquist frequency for capture of images used in super-resolution processing

Patent metadata

Field	Value
Publication number	US-9578237-B2
Application number	US-201514705919-A
Country	US
Kind code	B2
Filing date	May 6, 2015
Priority date	Jun 28, 2011
Publication date	Feb 21, 2017
Grant date	Feb 21, 2017

Abstract

A variety of optical arrangements and methods of modifying or enhancing the optical characteristics and functionality of these optical arrangements are provided. The optical arrangements being specifically designed to operate with camera arrays that incorporate an imaging device that is formed of a plurality of imagers that each include a plurality of pixels. The plurality of imagers include a first imager having a first imaging characteristics and a second imager having a second imaging characteristics. The images generated by the plurality of imagers are processed to obtain an enhanced image compared to images captured by the imagers. In many optical arrangements the MTF characteristics of the optics allow for contrast at spatial frequencies that are at least as great as the desired resolution of the high resolution images synthesized by the array camera, and significantly greater than the Nyquist frequency of the pixel pitch of the pixels on the focal plane, which in some cases may be 1.5, 2 or 3 times the Nyquist frequency.

First claim

What is claimed is:

1. An array camera, comprising:
a plurality of cameras, where each camera includes separate optics, and a plurality of light sensing elements;
a processor;
wherein the optics of each of the plurality of cameras are formed so that each camera has a field of view that is shifted with respect to the field-of-views of the other cameras and so that each shift includes a sub-pixel shifted view of the scene;
wherein the light sensing elements of a given camera in the plurality of cameras have a pixel pitch defining a camera Nyquist frequency, and where the optics of the given camera have a modular transfer function (MTF) such that the optics optically resolve contrast at spatial frequencies higher than the camera Nyquist frequency (Ny);
wherein the optics of each camera in the plurality of cameras comprise a five-surface optical arrangement comprising:
a first lens element having a first convex proximal surface and a first concave distal surface, wherein the diameter of the first convex surface is larger than the diameter of the first concave surface;
a second lens element having a second concave proximal surface and a second convex distal surface, wherein the diameter of the second concave proximal surface is smaller than the diameter of the second convex surface; and
a third lens element having a third concave proximal surface and a third planar distal surface, wherein the diameter of the third concave proximal surface is larger than the diameters of any of the surfaces of the first and second lens elements;
wherein the first, second and thirds lens elements are arranged sequentially in optical alignment with an imager positioned at the distal end thereof; and

wherein software directs the processor to:
obtain a set of low resolution images from the plurality of cameras, where each of the low resolution images include aliasing patterns;
determine disparity between pixels in the set of low resolution images to generate a depth map from a reference viewpoint, where the depth map indicates distances to surfaces of scene objects from the reference viewpoint;
synthesize a high resolution image using the set of images and the depth map, where the spatial frequency at which the high resolution image displays contrast is greater than the camera Nyquist frequencies (Ny) of the plurality of cameras and less than the spatial frequencies at which the optics of each camera optically resolve contrast.

2. The array camera of claim 1 , wherein the aliasing patterns in each image in the set of the low resolution images include differences due to the different sub-pixel shifted views of the scene provided by the optics of the plurality of cameras.

3. The array camera of claim 1 , wherein the software further directs the processor to synthesize a high resolution image by:
determining scene dependent geometric corrections to apply to the pixels from each of the images within the set of low resolution images to eliminate disparity; and
fusing the set of low resolution images using the scene dependent geometric corrections.

4. The array camera of claim 3 , wherein the software further directs the processor to perform super resolution processing to reconstruct the high resolution image using the fused image, the scene dependent geometric corrections, and the set of low resolution images.

5. The array camera of claim 1 , wherein the MTF of the optics of a given camera in the plurality of cameras is such that the optics optically resolve contrast at spatial frequencies at least 1.5 times the camera Nyquist frequency Ny.

6. The array camera of claim 1 , wherein the MTF of the optics a given camera in the plurality of cameras is such that the optics optically resolve contrast at spatial frequencies at least 2 times the camera Nyquist frequency Ny.

7. The array camera of claim 1 , wherein the MTF of the optics a given camera in the plurality of cameras is such that the optics optically resolve contrast at spatial frequencies at least 3 times the camera Nyquist frequency Ny.

8. The array camera of claim 1 , wherein the MTF of the optics a given camera in the plurality of cameras is such that the optics optically resolve contrast at spatial frequencies at least 10% greater than the camera Nyquist frequency Ny multiplied by the ratio of the resolution of the high resolution image to the resolution of the images in the set of low resolution images.

9. The array camera of claim 1 , wherein the MTF of the optics a given camera in the plurality of cameras is such that the optics optically resolve contrast at spatial frequencies at least 20% greater than the camera Nyquist frequency Ny multiplied by the ratio of the resolution of the high resolution image to the resolution of the images in the set of low resolution images.

10. The array camera of claim 1 , wherein the MTF of the optics a given camera in the plurality of cameras is such that the optics optically resolve contrast at spatial frequencies at least 30% greater than the camera Nyquist frequency Ny multiplied by the ratio of the resolution of the high resolution image to the resolution of the images in the set of low resolution images.

11. The array camera of claim 1 , wherein the camera array is a monolithic integrated module comprising a single semiconductor substrate on which all of the sensor elements are formed, and optics including a plurality of lens elements, where each lens element forms part of the separate optics for one of the cameras.

12. The array camera of claim 1 , wherein each of the cameras includes one of a plurality of different types of filter.

13. The array camera of claim 12 , wherein cameras having the same type of filter are uniformly distributed about the geometric center of the camera array.

14. The array camera of claim 1 , wherein cameras that include different types of filter operate with different operating parameters.

15. An array camera, comprising:
a plurality of cameras, where each camera includes separate optics, and a plurality of light sensing elements;
a processor;
wherein the optics of each of the plurality of cameras are formed so that each camera has a field of view that is shifted with respect to the field-of-views of the other cameras and so that each shift includes a sub-pixel shifted view of the scene;
wherein the light sensing elements of a given camera in the plurality of cameras have a pixel pitch defining a camera Nyquist frequency, and where the optics of the given camera have a modular transfer function (MTF) such that the optics optically resolve contrast at spatial frequencies higher than the camera Nyquist frequency (Ny);
wherein the optics of each camera in the plurality of cameras comprise a five-surface optical arrangement comprising:
a first lens element having a first convex proximal surface and a first concave distal surface, wherein the diameter of the first convex surface is larger than the diameter of the first concave surface;
a second lens element having a second concave proximal surface and a second convex distal surface, wherein the diameter of the second concave proximal surface is smaller than the diameter of the second convex surface; and
a third lens element having a third concave proximal surface and a third planar distal surface, wherein the diameter of the third concave proximal surface is larger than the diameters of any of the surfaces of the first and second lens elements;

wherein the first, second and thirds lens elements are arranged sequentially in optical alignment with an imager positioned at the distal end thereof; and
wherein software directs the processor to:
obtain a s

Assignees

• Pelican Imaging Corp
• Fotonation Cayman Ltd

Inventors

• Duparre Jacques
• Lelescu Dan
• Venkataraman Kartik

Classifications

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• G02B13/0035 Primary

  having three lenses · CPC title

• H04N23/951 Primary

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title