Auto-alignment of image sensors in a multi-camera system

Patent metadata

Field	Value
Publication number	US-9325917-B2
Application number	US-201514606019-A
Country	US
Kind code	B2
Filing date	Jan 27, 2015
Priority date	Mar 20, 2014
Publication date	Apr 26, 2016
Grant date	Apr 26, 2016

Abstract

A pair of cameras having an overlapping field of view is aligned based on images captured by image sensors of the pair of cameras. A pixel shift is identified between the images. Based on the identified pixel shift, a calibration is applied to one or both of the pair of cameras. To determine the pixel shift, the camera applies correlation methods including edge matching. Calibrating the pair of cameras may include adjusting a read window on an image sensor. The pixel shift can also be used to determine a time lag, which can be used to synchronize subsequent image captures.

First claim

What is claimed is:

1. A computer-implemented method for synchronizing a pair of image sensors, the method comprising:
capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;
correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels along the rolling shutter direction such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;
identifying, by one or more processors, a pixel shift between the captured images based on the first number of pixels; and
calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors.

2. The method of claim 1 , wherein correlating the captured image data representative of the overlapping field of view comprises:
compressing, for each image, pixels in the captured image data representative of the overlapping field of view; and
correlating the compressed pixels of each image.

3. The method of claim 1 , further comprising:
responsive to a determination that the captured images include less than a threshold amount of image data representative of the overlapping field of view required to correlate the captured image data, capturing an additional pair of images having an overlapping field of view with the image sensors.

4. The method of claim 1 , further comprising:
determining a time lag between the image sensors based on the identified pixel shift.

5. The method of claim 4 , wherein determining the time lag between the image sensors comprises determining an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

6. The method of claim 4 , wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:
initiating image capture at the first image sensor; and
after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

7. The method of claim 1 , wherein calibrating the at least one image sensor comprises:
adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

8. The method of claim 1 , wherein the pair of image sensors are controlled by a master camera controlling a slave camera.

9. The method of claim 1 , repeated to synchronize a plurality of pairs of image sensors.

10. A system comprising a pair of cameras, each camera having an image sensor, the system further comprising at least one processor and a non-transitory computer readable medium comprising instructions for synchronizing the image sensors, the instructions executable by the at least one processor, the instructions comprising instructions for:
capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;
correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels along the rolling shutter direction such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;
identifying, by one or more processors, a pixel shift between the captured images based on the first number of pixels; and
calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors.

11. The system of claim 10 , wherein calibrating the at least one image sensor comprises:
adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

12. The system of claim 10 , further comprising instructions for:
determining a time lag between the image sensors based, at least in part, on the identified pixel shift and an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

13. The system of claim 12 , wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:
initiating image capture at the first image sensor; and
after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

14. The system of claim 10 , further comprising a housing encompassing the pair of cameras.

15. The system of claim 10 , further comprising a third camera having an additional image sensor, the instructions further comprising instructions to synchronize the additional image sensor with the image sensors.

16. A non-transitory computer-readable medium comprising instructions for synchronizing a pair of image sensors, the instructions executable by at least one processor, the instructions comprising instructions for:
capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;
correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels along the rolling shutter direction such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;
identifying, by one or more processors, a pixel shift between the captured images based on the first number of pixels; and
calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors.

17. The computer-readable medium of claim 16 , wherein calibrating the at least one image sensor comprises:
adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

18. The computer-readable medium of claim 16 , further comprising instructions for:
determining a time lag between the image sensors based, at least in part, on the identified pixel shift and an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

19. The computer-readable medium of claim 18 , wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:
initiating image capture at the first image sensor; and
after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

20. The computer readable medium of claim 16 , the instructions further comprising instructions to synchronize an additional image sensor with the pair of image sensors.

Assignees

• Gopro Inc

Inventors

• Macmillan Timothy
• Campbell Scott Patrick
• Newman David A
• Sun Yajie

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N25/531

  by controlling rolling shutters in CMOS SSIS · CPC title

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title

• H04N17/002

  for television cameras · CPC title