Imaging arrangements and methods therefor

Patent metadata

Field	Value
Publication number	US-9479685-B2
Application number	US-201414573991-A
Country	US
Kind code	B2
Filing date	Dec 17, 2014
Priority date	Oct 1, 2004
Publication date	Oct 25, 2016
Grant date	Oct 25, 2016

Abstract

Image data is processed to facilitate focusing and/or optical correction. According to an example embodiment of the present invention, an imaging arrangement collects light data corresponding to light passing through a particular focal plane. The light data is collected using an approach that facilitates the determination of the direction from which various portions of the light incident upon a portion of the focal plane emanate from. Using this directional information in connection with value of the light as detected by photosensors, an image represented by the light is selectively focused and/or corrected.

First claim

What is claimed is:

1. A computer implemented method for processing digital image data, the method comprising:
receiving digital image data representing an image of a scene captured by a digital camera device, wherein the digital image data represents a plurality of light rays from the scene that arrive at a physical lens plane of the digital camera device via a physical aperture; and
generating a processed image of the scene by operating on the digital image data, comprising,
computing output pixels that make up the scene, comprising modeling a virtual aperture on a virtual lens plane such that the processed image appears as if it were captured using the virtual aperture;
computing a set of output images of the scene refocused at different focal depths within the scene;
for each pixel in the scene, determining a pixel from the set of output images having the highest local contrast; and
assembling the pixels having the highest local contrast into an output image of the scene with extended depth of field as compared to a depth of field of an image of the scene as captured by the digital camera device.

2. The method of claim 1 , wherein modeling a virtual aperture comprises using a virtual aperture function on the virtual lens plane.

3. The method of claim 2 , wherein the virtual aperture function comprises one or more scalar functions.

4. The method of claim 2 wherein the digital image data comprises light ray data, the method further comprising:
computing values of virtual points on a virtual film, comprising,
determining all light rays that start at different points on the virtual lens plane and converge on a point on the virtual film; and
applying a computational function to all of the light rays; and
summing all of the light rays.

5. The method of claim 4 , wherein the computational function comprises weighting the light rays by the virtual aperture function.

6. The method of claim 5 , further comprising varying the virtual aperture function from output pixel to output pixel.

7. The method of claim 6 , wherein the virtual aperture function is chosen to mask light rays from undesired regions of the scene.

8. The method of claim 6 , further comprising the processing circuitry receiving input from a user indicating virtual aperture parameters to be used.

9. The method of claim 6 , further comprising:
receiving user input to select a region of an output image of the scene;
receiving user input to select an image formation procedure;
receiving user input to alter virtual aperture parameters; and
generating an output image using the user input and the digital image data.

10. The method of claim 1 , wherein the digital camera device comprises a plurality of microlenses, and wherein the method further comprises:
extracting a photosensor value associated with each microlens, where each microlens is located a same relative position within an image on the microlens; and
generating an image of the scene with an extended depth of field as compared to a depth of field of an image of the scene as captured by the digital camera device.

11. A digital imaging system for generating a virtual image of a scene, wherein the virtual image includes a plurality of pixels, the system comprising:
a photosensor array, having a plurality of photosensors configured to detect light including a plurality of light rays from the scene, wherein each photosensor, in response to detecting the light incident thereon, generates light ray data;
an optics arrangement, having a physical focus, configured to direct light rays from the scene to the photosensor array via a physical focal plane, and
processing circuitry configurable to generate the virtual image, wherein the virtual image corresponds to an image as would be produced using a virtual lens plane and a virtual film surface, wherein generating the virtual image comprises modeling a substantially non-circular virtual aperture on the virtual lens plane such that the processed image appears as if it were captured using the virtual aperture.

12. The system of claim 11 , wherein the virtual aperture function comprises smoothly varying values implementing distinct regions in a virtual focal plane corresponding to the virtual aperture.

13. The system of claim 11 , wherein the virtual aperture function comprises smoothly varying values, including negative values, implementing distinct regions in a virtual focal plane corresponding to the virtual aperture.

14. The system of claim 11 , wherein the value of the pixel is computed with an arbitrary function that depends on values of the light rays.

15. The system of claim 14 , wherein the arbitrary function contains discontinuous program branches depending on the value of test functions computed on the values of the light rays.

16. The system of claim 11 , wherein the processing circuitry is configured to generate a virtual image of the scene by combining selected light rays based on aberrations in the optics arrangement.

17. The system of claim 16 , wherein the processing circuitry is configured to:
determine an angle of incidence of the detected light rays as characterized by the location of each photosensor detecting the light;
determine aberrated rays using the angle of incidence of the detected light rays;
re-sort the aberrated rays to correct for aberrations in the optics arrangement;
determine a weight for each light ray based on the virtual aperture function; and
generate the virtual image of the scene by combining selected aperture-weighted light rays and, and re-sorted aberrated light rays.

18. The system of claim 11 , wherein the processing circuitry is further configured to:
determine an angle of incidence of the detected light rays as characterized by the location of each photosensor detecting the light;
map the path of the light rays using the angle of incidence of the detected light rays and tracing of light rays through the optics arrangement;
determine aberrated rays using the mapped light rays;
re-sort the aberrated rays,
weight the re-sorted rays based on where they intersect the virtual aperture and the virtual aperture function, and
generate a virtual image of the scene by combining selected light rays and re-sorted aberrated rays to reduce the effect of aberrations in the optics arrangement.

19. A non-transient computer readable medium having stored thereon instructions that when executed by processing circuitry cause the execution of a method for processing digital image data, the method comprising:
receiving digital image data representing an image of a scene captured by a digital camera device, wherein the digital image data represents a plurality of light rays from the scene that arrive at a physical lens plane of the digital camera device via a physical aperture;
operating on the digital image data to generate a processed image of the scene, including computing output pixels that make up the scene, comprising modeling a virtual aperture on a virtual lens plane such that the processed image appears as if it were captured using the virtual aperture;
computing a set of output images of the scene refocused at different focal depths within the scene;
for each pixel in the scene, the processing circuity determining a pixel from the set of output images having the highest local contrast; and
assembling the pixels having the highest local contrast into an output image of the scene with extended depth of field as compared to a depth of field of an image of the scene as captured by the digital camera device.

Assignees

• Univ Leland Stanford Junior

Inventors

• Ng Yi-Ren
• Hanrahan Patrick
• Levoy Marc S
• Horowitz Mark A

Classifications

• G02B27/0075 Primary

  with means for altering, e.g. increasing, the depth of field or depth of focus · CPC title

• H04N23/951 Primary

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• G02B3/0056 Primary

  arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses (G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085) · CPC title

• H04N25/611

  Correction of chromatic aberration · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title