Capturing and processing of images including occlusions focused on an image sensor by a lens stack array

Patent metadata

Field	Value
Publication number	US-12563310-B2
Application number	US-202418597798-A
Country	US
Kind code	B2
Filing date	Mar 6, 2024
Priority date	May 20, 2008
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

Systems and methods for implementing array cameras configured to perform super-resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. An imaging device in accordance with one embodiment of the invention includes at least one imager array, and each imager in the array comprises a plurality of light sensing elements and a lens stack including at least one lens surface, where the lens stack is configured to form an image on the light sensing elements, control circuitry configured to capture images formed on the light sensing elements of each of the imagers, and a super-resolution processing module configured to generate at least one higher resolution super-resolved image using a plurality of the captured images.

First claim

What is claimed is:

1 . A method of generating a depth map from a plurality of images captured from different viewpoints, comprising:
capturing a plurality of images from different viewpoints, where each image includes pixels that are occluded in at least one other image from the plurality of images; normalizing the plurality of images based upon calibration data; measuring parallax information between the normalized images by comparing similarity of neighborhoods of pixels for different parallax-induced shifts; and generating a depth map using the measured parallax information.

2 . The method of claim 1 , wherein the plurality of images are captured by at least one camera array.

3 . The method of claim 2 , wherein the at least one camera array is a camera array of a mobile device.

4 . The method of claim 1 , wherein the plurality of images are captured by a plurality of cameras.

5 . The method of claim 4 , wherein the plurality of images include different occlusion sets, an occlusion set of a specific image captured by a given camera is a portion of a scene visible to a baseline camera from the plurality of cameras that is occluded from a viewpoint of the given camera; and wherein the plurality of cameras includes a first camera that captures pixels around an edge of a foreground object that is visible to the baseline camera and is in an occlusion set of a second of the plurality of cameras.

6 . The method of claim 4 , wherein cameras in the plurality of cameras are configured to operate with at least one difference in operating parameters.

7 . The method of claim 1 , wherein the plurality of images comprises a plurality of low resolution images, wherein the method further comprises generating at least one higher resolution super-resolved image using the plurality of the low resolution images by performing parallax detection to generate parallax information to align portions of the plurality of low resolution images.

8 . The method of claim 1 , wherein measuring parallax information comprises determining the parallax that yields a highest correlation between pixels from the plurality of images.

9 . The method of claim 1 , wherein measuring parallax information comprises calculating a parallax difference that yields a highest pixel correlation, wherein the parallax difference that yields the highest pixel correlation is determined by keeping track of various pair-wise measurements.

10 . The method of claim 1 , wherein measuring parallax information comprises performing pair-wise measurements of neighborhoods of pixels to determine pixel similarity for different parallax-induced shifts.

11 . The method of claim 10 , wherein measuring parallax information further comprises: determining a parallax that yields a highest similarity between pixels from images captured by keeping track of various pair-wise measurements of neighborhoods of pixels, and calculating a parallax that yields the highest similarity as a best least squares fit of the pair-wise measurements of neighborhoods of pixels.

12 . The method of claim 1 , further comprising generating at least one image by fusing aligned portions of the plurality of captured images using the depth map.

13 . The method of claim 12 , further comprising performing super-resolution processing on the fused image portions to synthesize a super-resolution image.

14 . The method of claim 1 , further comprising generating a higher resolution super-resolved image synthesized using the plurality of images and the parallax information to compensate for parallax in the plurality of images.

15 . The method of claim 1 , further comprising identifying occluded pixels based upon the measured parallax information.

16 . The method of claim 1 , wherein comparing the similarity of neighborhoods of pixels for different parallax-induced shifts comprises comparing the similarity of neighborhoods of pixels for different parallax-induced shifts along scan lines.

17 . The method of claim 1 , wherein normalizing the plurality of images based on calibration data comprises:
detecting and metering parallax using a parallax confirmation and measurement module, where detecting and metering parallax comprises ignoring pixels in images of the plurality of images that are in an exposed occlusion set; aligning portions of images in the plurality of images captured by different cameras to compensate for parallax using an image pixel correlation module based upon the detected and metered parallax and stored calibration data; and obtaining a higher resolution image having a resolution that is higher than resolutions of the images in the plurality of images using a super-resolution module, where color information around at least one edge of a foreground object that is visible in a baseline image and in an occlusion set of a second image is reconstructed in the higher resolution image using the pixels captured by a first image.

18 . The method of claim 1 , further wherein measuring parallax information comprises comparing a difference between average values of neighboring pixels with a threshold and flagging a potential presence of parallax when the difference exceeds the threshold.

Assignees

• Adeia Imaging Llc

Inventors

• Venkataraman Kartik
• Jabbi Amandeep S
• Mullis Robert H
• Duparre Jacques
• Hu Shane Ching-Feng

Classifications

• G02B3/0062

  Stacked lens arrays, i.e. refractive surfaces arranged in at least two planes, without structurally separate optical elements in-between · CPC title

• G02B3/0056

  arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses (G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085) · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title