Auto-alignment of image sensors in a multi-camera system

Patent metadata

Field	Value
Publication number	US-11375173-B2
Application number	US-202017061885-A
Country	US
Kind code	B2
Filing date	Oct 2, 2020
Priority date	Mar 20, 2014
Publication date	Jun 28, 2022
Grant date	Jun 28, 2022

Abstract

An image capture device having multiple image sensors having overlapping fields of view that aligns the image sensors based on images captured by image sensors. A pixel shift is identified between the images. Based on the identified pixel shift, a calibration is applied to one or more of the image sensors. To determine the pixel shift, a processor applies correlation methods including edge matching. Calibrating the image sensors may include adjusting a read window on an image sensor. The pixel shift can also be used to determine a time lag, which can be used to synchronize subsequent image captures.

First claim

What is claimed is:

1. An image capture device comprising:
a first image sensor having a first field of view, the first image sensor configured to capture a first image;
a second image sensor having a second field of view, wherein a portion of the first field of view and a portion of the second field of view comprise an overlapping field of view, the second image sensor configured to capture a second image; and
a processor configured to:
identify a pixel shift between the first image and the second image;
calibrate the first image sensor or the second image sensor iteratively based on the identified pixel shift until a next identified pixel shift is below a predefined pixel shift threshold; and
output a two-dimensional (2D) or a three-dimensional (3D) image based on the first image and the second image; and

an edge matching module that accesses edge information of the first image and the second image,
wherein the processor is configured to determine the pixel shift based on the edge information.

2. The image capture device of claim 1 , wherein the processor is further configured to:
compress, for each image, image pixels representative of the overlapping field of view, image portions of uniform color, and image portions of uniform texture; and
correlate the compressed pixels of each image.

3. The image capture device of claim 2 , wherein the processor is further configured to compress the image pixels in a direction perpendicular to a rolling shutter direction.

4. The image capture device of claim 1 , wherein the processor is configured to adjust a read window of the first image sensor in a rolling shutter direction to calibrate the first image sensor.

5. The image capture device of claim 4 , wherein the read window comprises a portion of the first image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

6. The image capture device of claim 1 , wherein the processor is configured to adjust a read window of the second image sensor in a rolling shutter direction to calibrate the second image sensor.

7. The image capture device of claim 6 , wherein the read window comprises a portion of the second image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

8. A method comprising:
obtaining a first image from a first sensor and a second image from a second sensor substantially simultaneously, wherein a portion of the second image overlaps with a portion of the first image;
identifying a pixel shift between the first image and the second image;
calibrating the first image sensor or the second image sensor iteratively based on the identified pixel shift until a next identified pixel shift is below a predefined pixel shift threshold;
smoothing the first image and the second image by removing artifacts and irregularities to reduce image data associated with the first image and the second image; and
outputting a two-dimensional (2D) or a three-dimensional (3D) image based on the first image and the second image.

9. The method of claim 8 further comprising:
compressing image pixels representative of the overlapping field of view; and
correlating the compressed pixels of each image.

10. The method of claim 9 , wherein the compressing image pixels includes compressing the image pixels in a direction perpendicular to a rolling shutter direction, and further includes cropping and removing portions of the first image and the second image that are smoothed.

11. The method of claim 8 , wherein calibrating the first image sensor includes adjusting a read window of the first image sensor in a rolling shutter direction.

12. The method of claim 11 , wherein the read window comprises a portion of the first image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

13. The method of claim 8 , wherein calibrating the second image sensor includes adjusting a read window of the second image sensor in a rolling shutter direction.

14. The method of claim 13 , wherein the read window comprises a portion of the second image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

15. A system comprising:
a first image sensor having a first field of view, the first image sensor configured to capture a first image;
a second image sensor having a second field of view, wherein a portion of the first field of view and a portion of the second field of view comprise an overlapping field of view, the second image sensor configured to capture a second image;
an edge matching module;
an edge smoothing module; and
a processor configured to:
correlate edges of the first image and the second image with the edge matching module;
identify a pixel shift between the first image and the second image based upon the correlated edges of the first image and the second image;
calibrate the first image sensor or the second image sensor iteratively based on the identified pixel shift until a next identified pixel shift is below a predefined pixel shift threshold;
smooth the first image and the second image to remove artifacts and other irregularities; and
output an image based on the first image and the second image.

16. The system of claim 15 , wherein the output image is a two-dimensional (2D) image.

17. The system of claim 15 , wherein the output image is a three-dimensional (3D) image.

18. The system of claim 15 , wherein the processor is further configured to compress the pixels in a direction perpendicular to a rolling shutter direction.

19. The system of claim 15 , wherein the processor is configured to adjust a read window of the first image sensor in a rolling shutter direction to calibrate the first image sensor.

20. The system of claim 15 , wherein the processor is configured to adjust a read window of the second image sensor in a rolling shutter direction to calibrate the second image sensor.

Assignees

• Gopro Inc

Inventors

• Macmillan Timothy
• Campbell Scott Patrick
• Newman David A
• Sun Yajie

Classifications

• H04N25/531

  by controlling rolling shutters in CMOS SSIS · CPC title

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N13/246 Primary

  Calibration of cameras · CPC title